From 2a66552cc006c48385126c98e85ffd96bf340ac8 Mon Sep 17 00:00:00 2001 From: Eric Timmons Date: Fri, 26 Jun 2020 16:45:06 -0400 Subject: [PATCH 1/8] Add Gitlab CI jobs for testing on a variety of Lisps The jobs that use the Makefile based test harness are created unconditionally. There are additionally jobs that use the Lisp scripting based harness which are added only if the CI environment variable USE_ASDF_TOOLS is set. --- README.md | 31 ++++++++ gitlab-ci.yml | 195 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 226 insertions(+) create mode 100644 gitlab-ci.yml diff --git a/README.md b/README.md index dd91edda..f0091062 100644 --- a/README.md +++ b/README.md @@ -144,6 +144,34 @@ indeed, a developer may not only make mistakes, but may deliberately introduce or re-introduce bugs at some place to test code in another place. +Contributing to ASDF +-------------------- + +Bugs can be filled on ASDF by reporting them on the [Gitlab issue +tracker](https://gitlab.common-lisp.net/asdf/asdf/-/issues) or sending them to +the [asdf-devel mailing list](https://mailman.common-lisp.net/listinfo/asdf-devel). + +You can contribute code to ASDF development by forking the +[repository](https://gitlab.common-lisp.net/asdf/asdf) and sending a merge +request or sending a patch to the asdf-devel mailing list. + +If you fork the repository on Gitlab, note that Gitlab CI is enabled to help in +automated testing. While not exhaustive, this can help make sure you don't +inadvertantly break anything with your patch! The tests will be run any time +you submit a merge request or manually trigger a run using Gitlab's UI. In +order for the tests to run properly (namely the ASDF upgrade tests), you must +ensure your fork contains the tags for every released version of ASDF. If your +fork is freshly created, this will happen automatically. However, if there has +been a release since you forked, you need to update your tags. Assuming that +your fork is the `origin` remote and upstream is the `upstream` remote, you can +do this by running: + + git fetch upstream --tags + git push origin --tags + +If you would like to enable test jobs that use the Lisp scripting test harness, +set the variable `ENABLE_ASDF_TOOLS` on a pipeline. + Debugging ASDF -------------- @@ -297,3 +325,6 @@ How do I navigate this source tree? * [TODO](TODO) * Plenty of ideas for how to further improve ASDF (not all of them guaranteed good ideas.) + +* [gitlab-ci.yml](gitlab-ci.yml) + * A YAML file describing jobs for Gitlab CI to run. diff --git a/gitlab-ci.yml b/gitlab-ci.yml new file mode 100644 index 00000000..944022e7 --- /dev/null +++ b/gitlab-ci.yml @@ -0,0 +1,195 @@ +############################################################################### +# Global configuration +############################################################################### + +variables: + # Just let the runner fetch and update the submodules for us. + GIT_SUBMODULE_STRATEGY: normal + + +# This causes pipelines to be created only on the default branch, tags, merge +# requests, and when triggered via the web interface. +workflow: + rules: + - if: $CI_MERGE_REQUEST_IID + - if: $CI_COMMIT_TAG + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_PIPELINE_SOURCE == "web" + +############################################################################### +# Build jobs +############################################################################### + +Build ASDF: + stage: build + image: debian:latest + script: + - apt-get update + - apt-get install -y --no-install-recommends git make + - make + artifacts: + paths: + - build/asdf.lisp + +Build docs: + stage: build + image: debian:latest + script: + - apt-get update + - apt-get install -y --no-install-recommends git make texinfo texlive + - make -C doc + artifacts: + paths: + - doc/asdf.html + - doc/asdf.info + - doc/asdf.pdf + +Build asdf-tools: + stage: build + image: daewok/sbcl:latest + script: + - apt-get update + - apt-get install -y --no-install-recommends git make + - make -f Makefile-lisp-scripting build-asdf-tools + artifacts: + paths: + - build/asdf-tools + rules: + - if: $ENABLE_ASDF_TOOLS + +############################################################################### +# Testing Templates +############################################################################### + +.Regression tests: + stage: test + image: daewok/$l:latest + script: + - apt-get update + - apt-get install -y --no-install-recommends git make + - make + - make test + artifacts: + paths: + - build/results/$l-test.text + needs: [] + +.Upgrade tests: + stage: test + image: daewok/$l:latest + script: + - apt-get update + - apt-get install -y --no-install-recommends git make + - make + - make test-upgrade + artifacts: + paths: + - build/results/$l-upgrade.text + needs: [] + +.Scripting regression tests: + extends: .Regression tests + before_script: + - ln -s Makefile-lisp-scripting GNUmakefile + needs: + - Build asdf-tools + rules: + - if: $ENABLE_ASDF_TOOLS + +.Scripting upgrade tests: + extends: .Upgrade tests + before_script: + - ln -s Makefile-lisp-scripting GNUmakefile + needs: + - Build asdf-tools + rules: + - if: $ENABLE_ASDF_TOOLS + +############################################################################### +# Actual test jobs - Makefile based harness +############################################################################### + +ABCL regression tests: + extends: .Regression tests + variables: + l: abcl + +ABCL upgrade tests: + extends: .Upgrade tests + variables: + l: abcl + +CCL regression tests: + extends: .Regression tests + variables: + l: ccl + +CCL upgrade tests: + extends: .Upgrade tests + variables: + l: ccl + +ECL regression tests: + extends: .Regression tests + variables: + l: ecl + +ECL upgrade tests: + extends: .Upgrade tests + variables: + l: ecl + +SBCL regression tests: + extends: .Regression tests + variables: + l: sbcl + +SBCL upgrade tests: + extends: .Upgrade tests + variables: + l: sbcl + + +############################################################################### +# Actual test jobs - lisp scripting based harness +############################################################################### + +ABCL scripting regression tests: + extends: .Scripting regression tests + variables: + l: abcl + +ABCL scripting upgrade tests: + extends: .Scripting upgrade tests + variables: + l: abcl + +CCL scripting regression tests: + extends: .Scripting regression tests + variables: + l: ccl + +CCL scripting upgrade tests: + extends: .Scripting upgrade tests + variables: + l: ccl + +ECL scripting regression tests: + extends: .Scripting regression tests + variables: + l: ecl + +ECL scripting upgrade tests: + extends: .Scripting upgrade tests + variables: + l: ecl + +SBCL scripting regression tests: + extends: .Scripting regression tests + variables: + l: sbcl + +SBCL scripting upgrade tests: + extends: .Scripting upgrade tests + variables: + l: sbcl -- GitLab From 4ed687c9e3d8b88542163b472408eb5f66cceb24 Mon Sep 17 00:00:00 2001 From: Eric Timmons Date: Sun, 28 Jun 2020 23:35:40 -0400 Subject: [PATCH 2/8] Fix run_upgrade_tests function in run-tests.sh If an upgrade test failed, that function never exited with a non-zero status. While exit 1 was called, it was in a pipeline, so the exit code was set as the exit code of the last command in the pipeline (tee) which always exited 0. Switch to keeping the status in a separate file (like run_tests). A local variable cannot be used, because the pipeline also makes the block containing the for loop run in a sub shell. --- test/run-tests.sh | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/test/run-tests.sh b/test/run-tests.sh index 4a8ca294..34a3a409 100755 --- a/test/run-tests.sh +++ b/test/run-tests.sh @@ -476,6 +476,7 @@ run_upgrade_tests () { su=test/script-support.lisp tags="`upgrade_tags`" methods="`upgrade_methods`" + echo success > build/results/${lisp}-upgrade.status { for tag in $tags ; do for method in $methods ; do @@ -491,10 +492,21 @@ run_upgrade_tests () { echo "$icmd" echo "then copy/paste:" echo "(load \"$su\") (asdf-test::da) (test-upgrade $method \"$tag\")" - exit 1 ;} - fi ; done ; done - echo "Upgrade test succeeded for ${lisp}" + echo "failure" > "build/results/${lisp}-upgrade.status" ;} + fi ; done ; done ; + read status < "build/results/${lisp}-upgrade.status" + if [ "$status" = "success" ]; then + echo "Upgrade test succeeded for ${lisp}" + else + echo "Upgrade test failed for ${lisp}" + fi ; } 2>&1 | tee build/results/${lisp}-upgrade.text + # We need to reread the status because the piping to tee causes everything + # in the block above to be run in a subshell. + read status < "build/results/${lisp}-upgrade.status" + if [ "$status" = "failure" ]; then + exit 1 + fi } run_tests () { create_config -- GitLab From 3c018492a030b1a9409657799f082d9ca15e05be Mon Sep 17 00:00:00 2001 From: Eric Timmons Date: Wed, 14 Oct 2020 22:32:17 -0400 Subject: [PATCH 3/8] Switch to using images hosted by mitmers instead of daewok They are the same images, just moved to a new location. --- gitlab-ci.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/gitlab-ci.yml b/gitlab-ci.yml index 944022e7..3a422550 100644 --- a/gitlab-ci.yml +++ b/gitlab-ci.yml @@ -46,7 +46,7 @@ Build docs: Build asdf-tools: stage: build - image: daewok/sbcl:latest + image: mitmers/sbcl:latest script: - apt-get update - apt-get install -y --no-install-recommends git make @@ -63,7 +63,7 @@ Build asdf-tools: .Regression tests: stage: test - image: daewok/$l:latest + image: mitmers/$l:latest script: - apt-get update - apt-get install -y --no-install-recommends git make @@ -76,7 +76,7 @@ Build asdf-tools: .Upgrade tests: stage: test - image: daewok/$l:latest + image: mitmers/$l:latest script: - apt-get update - apt-get install -y --no-install-recommends git make -- GitLab From 62af6dd4db1131daf212bd4458915e413527fe6d Mon Sep 17 00:00:00 2001 From: Eric Timmons Date: Thu, 15 Oct 2020 13:16:24 -0400 Subject: [PATCH 4/8] Increase sleeps in test-run-program-unix When testing CCL using cl.net's Gitlab CI runners, the current sleep times do not appear to be sufficient. --- test/test-run-program-unix.script | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/test/test-run-program-unix.script b/test/test-run-program-unix.script index 15114142..b261b453 100644 --- a/test/test-run-program-unix.script +++ b/test/test-run-program-unix.script @@ -480,7 +480,7 @@ (let ((process-info (launch-program "./sleeper.sh 12 1"))) (handle-result "alive?" (process-alive-p process-info)) - (sleep 2) + (sleep 5) (handle-result "dead?" (not (process-alive-p process-info))) (wait-process process-info))) @@ -552,7 +552,7 @@ (let* ((process-info (launch-program "./sleeper.sh 12 1")) (status (process-status process-info))) (handle-result "running?" (equal :running status)) - (sleep 2) + (sleep 5) (multiple-value-bind (status exit-code) (process-status process-info) (handle-result "exited?" (equal :exited status)) @@ -568,7 +568,7 @@ #+(and lispworks (not lispworks7+)) (skip-test "Known to fail") (let* ((process-info (launch-program "./killer.sh"))) - (sleep 1) + (sleep 5) (multiple-value-bind (status code) (process-status process-info) (handle-result "exited or signaled?" @@ -588,7 +588,7 @@ (define-test ":exited (it exited >128; repeated query)" (let* ((process-info (launch-program "./killercontainer.sh"))) - (sleep 1) + (sleep 5) (multiple-value-bind (status code) (process-status process-info) (handle-result "exited?" (eq status :exited)) -- GitLab From 5bc681b9472b864cc902833d6e3cbc595e531693 Mon Sep 17 00:00:00 2001 From: Eric Timmons Date: Thu, 15 Oct 2020 17:07:59 -0400 Subject: [PATCH 5/8] Make Docker image tags configurable Default to latest, but use SBCL 2.0.8 for the moment as 2.0.9 has a bug that is triggered on cl.net's Gitlab CI Runners. This bug was likely fixed in https://github.com/sbcl/sbcl/commit/d0243a9f9961f0afdc09b555821b88edb2488be9 and is being backported to mitmers/sbcl:2.0.9. --- gitlab-ci.yml | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/gitlab-ci.yml b/gitlab-ci.yml index 3a422550..ea763047 100644 --- a/gitlab-ci.yml +++ b/gitlab-ci.yml @@ -63,7 +63,9 @@ Build asdf-tools: .Regression tests: stage: test - image: mitmers/$l:latest + image: mitmers/$l:$TAG + variables: + TAG: latest script: - apt-get update - apt-get install -y --no-install-recommends git make @@ -76,7 +78,9 @@ Build asdf-tools: .Upgrade tests: stage: test - image: mitmers/$l:latest + image: mitmers/$l:$TAG + variables: + TAG: latest script: - apt-get update - apt-get install -y --no-install-recommends git make @@ -143,11 +147,13 @@ SBCL regression tests: extends: .Regression tests variables: l: sbcl + TAG: 2.0.8 SBCL upgrade tests: extends: .Upgrade tests variables: l: sbcl + TAG: 2.0.8 ############################################################################### -- GitLab From 3b6d89205fdd54816f8416f8d2dbeaf079862f74 Mon Sep 17 00:00:00 2001 From: Eric Timmons Date: Thu, 15 Oct 2020 17:11:33 -0400 Subject: [PATCH 6/8] Allow upgrade jobs to fail Upgrades are currently broken on ABCL. Additionally, it seems the Jenkins test suite didn't even run them before. --- gitlab-ci.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/gitlab-ci.yml b/gitlab-ci.yml index ea763047..382acf4a 100644 --- a/gitlab-ci.yml +++ b/gitlab-ci.yml @@ -90,6 +90,7 @@ Build asdf-tools: paths: - build/results/$l-upgrade.text needs: [] + allow_failure: true .Scripting regression tests: extends: .Regression tests -- GitLab From 6b21e278612da82ad9af05679ef301cbe452deb7 Mon Sep 17 00:00:00 2001 From: Eric Timmons Date: Thu, 15 Oct 2020 17:34:03 -0400 Subject: [PATCH 7/8] Split regression and upgrade tests into separate stages --- gitlab-ci.yml | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/gitlab-ci.yml b/gitlab-ci.yml index 382acf4a..8ef639bd 100644 --- a/gitlab-ci.yml +++ b/gitlab-ci.yml @@ -16,6 +16,14 @@ workflow: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - if: $CI_PIPELINE_SOURCE == "web" +# Split regression testing into a separate stage than upgrade tests. The +# upgrade tests tend to run slower and are generally less important than +# regression tests for most changes. +stages: + - build + - regression + - upgrade + ############################################################################### # Build jobs ############################################################################### @@ -62,7 +70,7 @@ Build asdf-tools: ############################################################################### .Regression tests: - stage: test + stage: regression image: mitmers/$l:$TAG variables: TAG: latest @@ -77,7 +85,7 @@ Build asdf-tools: needs: [] .Upgrade tests: - stage: test + stage: upgrade image: mitmers/$l:$TAG variables: TAG: latest -- GitLab From 5a15171a44310fbf3ded80a93b4b7b6bdccaf04f Mon Sep 17 00:00:00 2001 From: Eric Timmons Date: Thu, 15 Oct 2020 17:58:12 -0400 Subject: [PATCH 8/8] Switch back to latest version of SBCL The necessary change has been backported to mitmers/sbcl:2.0.9. --- gitlab-ci.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/gitlab-ci.yml b/gitlab-ci.yml index 8ef639bd..3b9bd7d2 100644 --- a/gitlab-ci.yml +++ b/gitlab-ci.yml @@ -156,13 +156,11 @@ SBCL regression tests: extends: .Regression tests variables: l: sbcl - TAG: 2.0.8 SBCL upgrade tests: extends: .Upgrade tests variables: l: sbcl - TAG: 2.0.8 ############################################################################### -- GitLab