Unverified Commit 60502e8b authored by Eric Timmons's avatar Eric Timmons
Browse files

Update for common-lisp.dev and add new FAQS for GitLab CI

parent 06fbce24
......@@ -17,5 +17,15 @@ please follow the steps outlined in
### Deploy project pages
Use GitLab Pages to deploy your project pages
to `https://common-lisp.net/project/<project>/`
to `https://<project>.common-lisp.dev/`
using our [deployment instructions](/faq/using-gitlab-deploy-project-pages).
### Test a project
Use GitLab CI to test against multiple CL implementations using
our [GitLab CI for CL intro](/faq/using-gitlab-ci-for-cl).
### Populate a project page with content from multiple repos
Use GitLab CI to create multi-project pipelines using our
[multi-project CI instructions](/faq/using-gitlab-ci-multi-project-pipelines).
title: Using GitLab CI to test CL projects -- Common-Lisp.net FAQ
---
{table-of-contents :depth 2 :start 2 :label "Table of contents" }
# Using GitLab CI to test CL projects
This page contains instructions on how to leverage GitLab CI to test your CL
project against multiple Common Lisp implementations.
[GitLab CI](https://docs.gitlab.com/ee/ci/) is a very flexible piece of
software. As such, there is no One True Way of doing things. However, we
recommend that you start your GitLab CI journey using GitLab CI templates from
the CLCI project.
The [CLCI project](https://clci.common-lisp.dev/) both gathers links to a
variety of CI solutions for CL and develops templates for GitLab CI. The GitLab
CI template documentation can be found
at <https://clci.common-lisp.dev/clci-gitlab-ci/>. Its project repo is
at [clci/gitlab-ci](https://gitlab.common-lisp.net/clci/gitlab-ci/).
## Step 1: Write tests
In order to test your code, you must first write some tests! There are a wide
variety of Common Lisp testing frameworks. For a recent review of them,
see <https://sabracrolleton.github.io/testing-framework>.
Once you have chosen a testing framework and written tests, you should tie it
into ASDF such that `(asdf:test-system "foo")` will run your tests. You
framework should have instructions on how to do this. If it does not, see the
[ASDF best practices](https://gitlab.common-lisp.net/asdf/asdf/blob/master/doc/best_practices.md#trivial-testing-definition)
for guidance.
Additionally, it is
[best practice](https://gitlab.common-lisp.net/asdf/asdf/blob/master/doc/best_practices.md#testing-a-system)
for your system's test operation to signal an error if tests fail.
## Step 2: Write a script to run your tests
There are a large number of ways to write system definitions and run tests. So
CLCI's GitLab templates do not attempt to guess how you want your tests to be
run. Instead, you must provide a Common Lisp file that, when loaded with
`cl:load` will run your tests and either signal an error on failure or exit the
process with a non-zero exit code.
The templates default to looking for this script at `scripts/ci-test.lisp` in
your repo. However, it can be overridden by setting the `$CLCI_TEST_SCRIPT`
variable.
In the default configuration, this script will be loaded into a CL process that
already has ASDF loaded (defaulting to the latest version), and your chosen
dependency manager loaded (if any). Thus, for most projects the contents of
this file will simply be:
```common-lisp
(ql:quickload "foo/test")
(asdf:test-system "foo")
```
## Step 3: Add continuous integration through `.gitlab-ci.yml`
Next instructions have to be added through GitLab's CI (Continuous Integration)
platform to actually run your tests. The example below is a simple way to run
your tests on up to eight CL implementations.
```yaml
include:
project: 'clci/gitlab-ci'
ref: v2-stable
file:
- guarded-linux-test-pipeline.gitlab-ci.yml
variables:
# Uncomment if you want to use Quicklisp as your dependency manager during
# tests.
#
# CLCI_DEPENDENCY_MANAGER: "quicklisp"
#
# Uncomment if you have Git submodules that you want the runner to
# automatically init and update for you. Submodules are sometimes used by
# projects to bundle their test dependencies.
#
# GIT_SUBMODULE_STRATEGY: recursive
#
# Uncomment these lines if you want to test against Allegro, you have read
# the Allegro express license
# <https://franz.com/ftp/pub/legal/ACL-Express-20170301.pdf>, *and* your use
# of Allegro Express does not violate the license. Alternatively, uncomment
# and provide your own Docker image (or runner) that has Allegro installed
# with your actual license.
#
# CLCI_TEST_ALLEGRO: "yes"
# I_AGREE_TO_ALLEGRO_EXPRESS_LICENSE: "yes"
#
# Uncomment if you want to test on Clasp. It is disabled by default due to
# significant upstream churn. It will be enabled by default after their next
# release.
#
# CLCI_TEST_CLASP: "yes"
# This section is not strictly required, but prevents Gitlab CI from launching
# multiple redundent pipelines when a Merge Request is opened.
workflow:
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
- if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
when: never
- if: '$CI_COMMIT_BRANCH'
- if: '$CI_COMMIT_TAG'
```
Be sure to read the comments and uncomment lines that apply to you.
The above uses the pipeline defined by the
`guarded-linux-test-pipeline.gitlab-ci.yml` file to test against up to eight
(at the time of writing) CL implementations:
- ABCL
- Allegro
- CCL
- Clasp
- CLISP
- CMUCL
- ECL
- SBCL
By default, the template uses Docker images from the
[cl-docker-images project](https://cl-docker-images.common-lisp.dev/). These
images contain a precompiled CL implementation, as well as a number of commonly
used foreign libraries.
The `workflow` section has nothing to do with CL at all. GitLab CI treats
pipelines associated with Merge Requests differently than pipelines associated
with a branch. A poor GitLab CI configuration would spawn two pipelines for
Merge Requests: one associated with the MR and one associated with the
branch. This workflow tells GitLab CI to not create a "branch pipeline" once a
MR has been opened for the branch.
## Step 4: Deploy and test your new pipelines
If the syntax of your `.gitlab-ci.yml` file is correct, a new build will run on
every push.
Note that if you want to check that your `.gitlab-ci.yml` file has
valid syntax, you can use GitLab's "CI Lint" application. This application
is available in every project on the "Pipelines" page. For the "clo/cl-site"
project, it's located at https://gitlab.common-lisp.net/clo/cl-site/-/ci/lint
Please note that the CLCI GitLab templates have more functionality than just
testing. You can easily add pipelines to cut releases of your project when you
push a tag or periodically update your pinned dependencies if you use
CLPM. Additionally, you can easily add jobs that run arbitrary CL scripts
besides the one you wrote in `scripts/ci-test.lisp`
For more information, please make sure to
read <https://docs.gitlab.com/ce/user/project/pages/#overview>
and <https://clci.common-lisp.dev/clci-gitlab-ci/>.
title: Using GitLab CI Multi Project Pipelines -- Common-Lisp.net FAQ
---
{table-of-contents :depth 2 :start 2 :label "Table of contents" }
# Using GitLab CI Multi Project Pipelines
This page contains instructions on how to leverage GitLab CI to run pipelines
in multiple GitLab projects (repositories). This is particularly useful for
projects that have multiple repositories and/or automatically generated
documentation that should be kept up-to-date on the project website.
[GitLab CI](https://docs.gitlab.com/ee/ci/) is a very flexible piece of
software. As such, there is no One True Way of doing things. However, the
recipe below is being successfully used by the cl-tar project to keep its
website up-to-date with automatically generated documentation whenever it
performs a release.
We will focus on the interaction
between [cl-tar/cl-tar-file](https://gitlab.common-lisp.net/cl-tar/cl-tar-file)
and
[cl-tar/cl-tar.common-lisp.dev](https://gitlab.common-lisp.net/cl-tar/cl-tar.common-lisp.dev).
## Step 1: Generate documentation
First, generate your documentation using GitLab CI. The relevant job from the
`cl-tar-file` project is:
```yaml
# Build the docs and upload them to Gitlab.
generate docs:
extends:
- .clci sbcl
- .clci clpm script
variables:
CLCI_SCRIPT: scripts/generate-docs.lisp
artifacts:
paths:
- docs/
```
This job "extends" some helper jobs provided by
the [CLCI GitLab templates](https://clci.common-lisp.dev/clci-gitlab-ci/). The
`.clci sbcl` template says that this job should use SBCL. The `.clci clpm
script` template says that the latest version of ASDF should be installed, the
latest version of CLPM should be installed, the dependencies specified in the
CLPM lock file should be installed, and then the file pointed to by
`$CLCI_SCRIPT` should be `cl:load`ed into an SBCL process where ASDF has been
loaded and all the dependencies are availble.
The `scripts/generate-docs.lisp` file uses a CL-based documentation generator
to output docs to the `docs` folder.
Last, the `docs` folder is uploaded to GitLab as an artifact.
## Step 2: Trigger a pipeline in your website project
Next, you must trigger a job in your project's website's repo. The relevant job
from the `cl-tar-file` project is:
```yaml
# Notify the project containing the website that new documentation is ready to
# be fetched.
publish docs:
rules:
- if: '$CI_COMMIT_TAG =~ /^v[0-9]+(\.[0-9]+)*(-.*)?$/'
needs:
- "blocker:release:clci"
- "generate docs"
trigger: cl-tar/cl-tar.common-lisp.dev
variables:
CL_TAR_FILE_TAG: $CI_COMMIT_TAG
CL_TAR_FILE_PIPELINE: $CI_PIPELINE_ID
```
This job's `rules` ensure that it is only run when a tag is pushed to the repo
that looks like a version number.
The `needs` ensures that this job is not run until both the documentation has
actually been generated *and* all sanity checks performed by the CLCI GitLab
template release pipeline have finished. These sanity checks include making
sure the tag is protected and all tests pass.
The `trigger` says that a new pipeline should be trigged on the default branch
of the `cl-tar/cl-tar.common-lisp.dev` project.
Last, the `variables` section passes the pipeline ID and tag to the newly
created pipeline.
## Step 3: Download the artifact to the website project
Next, the newly triggered pipeline downloads and commits the generated
documentation. The relevant jobs from the `cl-tar.common-lisp.dev` project are
this one from `.gitlab-ci.yml`:
```yaml
download cl-tar-file docs:
variables:
CL_TAR_FILE_PIPELINE_ID: $CL_TAR_FILE_PIPELINE
CL_TAR_FILE_TAG: $CL_TAR_FILE_TAG
rules:
- if: $CL_TAR_FILE_PIPELINE
trigger:
include: .gitlab-ci.cl-tar-file.yml
strategy: depend
```
And this one from `.gitlab-ci.cl-tar-file.yml`:
```yaml
download docs:
image: alpine:3.13
script:
- apk add jq unzip git curl
- git config --global user.email "$GITLAB_USER_EMAIL"
- git config --global user.name "$GITLAB_USER_NAME"
- git checkout $CI_COMMIT_BRANCH
- git reset --hard origin/$CI_COMMIT_BRANCH
# Find the job id of the docs job
- JOB_ID="$(curl -fsSL "$CI_API_V4_URL/projects/1508/pipelines/$CL_TAR_FILE_PIPELINE_ID/jobs" | jq '.[] | select (.name == "generate docs") | .id')"
# Download the artifact.zip
- curl -fsSL "https://gitlab.common-lisp.net/cl-tar/cl-tar-file/-/jobs/$JOB_ID/artifacts/download?file_type=archive" > artifacts.zip
- unzip artifacts.zip
- mkdir -p cl-tar-file
- mv docs cl-tar-file/$CL_TAR_FILE_TAG
- git add cl-tar-file/$CL_TAR_FILE_TAG
- git commit -m "cl-tar-file $CL_TAR_FILE_TAG docs"
- git push https://git:$UPDATER_ACCESS_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH.git $CI_COMMIT_BRANCH:$CI_COMMIT_BRANCH
```
The first job simple creates a child pipeline from the
`.gitlab-ci.cl-tar-file.yml` if certain conditions are met (namely, the
variable `$CL_TAR_FILE_PIPELINE` is set). This child pipeline is completely
optional: the following job could have been written in `.gitlab-ci.yml` with
appropriate `rules` added.
The second job takes the pipeline ID that generated the documentation, queries
the GitLab API to find the URL for the artifacts, and then downloads the
artifacts. It then unzips them and copies the folder to the correct place in
the repo. Last, it commits it and pushes.
## Step 4: Build and deploy the site
After the push from Step 3, another pipeline is kicked off. This time, the new
documentation from the `cl-tar-file` project is present in the repo. This
triggers the following job in `cl-tar.common-lisp.dev`, which in turn ends up
building the website and deploying it using GitLab Pages.
```yaml
publish docs:
rules:
- if: '$CL_TAR_PIPELINE == null && $CL_TAR_FILE_PIPELINE == null'
trigger:
include: .gitlab-ci.publish.yml
strategy: depend
```
......@@ -6,7 +6,7 @@ title: Using GitLab to deploy project pages -- Common-Lisp.net FAQ
# Using GitLab to deploy project pages
This page contains instructions on how to deploy your project pages
(those hosted on 'https://common-lisp.net/project/<your-project>`).
(those hosted on `https://<your-project>.common-lisp.dev`).
Using deployment through GitLab, you can use your preferred site
generator in your preferred setup: from "simply copy the repository
......@@ -15,11 +15,10 @@ are supported by our common-lisp.net setup.
## Step 1: Create a repository for your project pages
In order to deploy your project pages to
`https://common-lisp.net/project/<project>`, you need a
project in GitLab named `<project>-site` in the group named
`<project>`. That is, the resulting repository will be viewable
at the URL `https://gitlab.common-lisp.net/<project>/<project>-site`.
In order to deploy your project pages to `https://<project>.common-lisp.dev`,
you need a project in GitLab named `<project>.common-lisp.dev` in the group
named `<project>`. That is, the resulting repository will be viewable at the
URL `https://gitlab.common-lisp.net/<project>/<project>.common-lisp.dev`.
## Step 2: Add content to the repository
......@@ -34,10 +33,10 @@ project. It holds the content to be deployed in a `src/` subdirectory
which can simply be renamed to `public/`. No further processing is
required.
More complex build steps can be found in the [clo/
cl-site](https://gitlab.common-lisp.net/clo/cl-site/) and [quickref/
quickref](https://gitlab.common-lisp.net/quickref/quickref/) projects
respectively.
More complex build steps can be found in
the [clo/cl-site](https://gitlab.common-lisp.net/clo/cl-site/) and the
[cl-tar/cl-tar.common-lisp.dev](https://gitlab.common-lisp.net/cl-tar/cl-tar.common-lisp.dev/)
projects respectively.
## Step 3: Add continuous integration through `.gitlab-ci.yml`
......@@ -53,17 +52,20 @@ deploying static site content without processing.
artifacts:
paths:
- public
tags:
- site-gen
only:
- master
The above defines the special purpose `pages` build job known to
GitLab to be associated with GitLab Pages deployment. GitLab CI
jobs can be tied to various "stages". The `pages` job should be
tied to the "deploy" stage. And last but not least, the job creates
artifacts in the `public` path which GitLab Pages uses to deploy
the site.
The above defines the special purpose `pages` build job known to GitLab to be
associated with GitLab Pages deployment.
GitLab CI provides two ways to order jobs. First, jobs can be tied to various
"stages" and jobs in any given stage are executed only after the previous
stages have completed. Alternatively, jobs can specify a list of "needs" and a
job can be executed once its needs are met, regardless if previous stages have
finished. In this example, the `pages` job is added to the "deploy" stage.
The `pages` job must upload the `public` folder as an artifact. Whatever is
present in the `public` folder is served using GitLab Pages.
To prevent every intermediate result on every branch from being
deployed to your project site, the job is restricted to building
......@@ -74,26 +76,16 @@ from the `master` branch only, using the
fragment.
The `tags` fragment mentioned below, links the job to CI builders
("runners"). Builders can only build jobs for which the job at
least one of the tags the builder has as well. On the common-lisp.net
infrastructure, this means that a job with the tag `site-gen` can
only be built with the builder tagged with the `site-gen` tag too.
tags:
- site-gen
Then, last but not least, the fragment `script: mv src public`
instructs the CI system to perform the "build" for this repository
(i.e. nothing more than to rename `src` to `public`).
As mentioned in the previous section, other, more complex, setups
and build instructions are possible. See for examples the [clo/
cl-site](https://gitlab.common-lisp.net/clo/cl-site/) and [quickref/
quickref](https://gitlab.common-lisp.net/quickref/quickref/) projects.
Note however, that these projects don't deploy to a `/project/<project>`
page (but to `https://common-lisp.net` and `https://quickref.common-lisp.net`
instead).
cl-site](https://gitlab.common-lisp.net/clo/cl-site/) and [cl-tar/
cl-tar.common-lisp.dev](https://gitlab.common-lisp.net/cl-tar/cl-tar.common-lisp.dev/) projects.
Note however, that `clo/cl-site` doesn't deploy to a `<project>.common-lisp.dev/`
page (but to `https://common-lisp.net` instead).
## Step 4: Deploy and test your new project pages
......
......@@ -30,9 +30,9 @@ title: Project Hosting
<p id="project-pages">Projects using GitLab for their version control, can take advantage of
<a href="https://docs.gitlab.com/ee/user/project/pages/">GitLab Pages</a>
to host their project site. An example of such a project is the <a
href="https://gitlab.common-lisp.net/boston-lisp/boston-lisp-site/">Boston Lisp
href="https://gitlab.common-lisp.net/boston-lisp/boston-lisp.common-lisp.dev/">Boston Lisp
site repository</a>; see the <a
href="https://gitlab.common-lisp.net/boston-lisp/boston-lisp-site/-/blob/master/.gitlab-ci.yml">.gitlab-ci.yml</a>
href="https://gitlab.common-lisp.net/boston-lisp/boston-lisp.common-lisp.dev/-/blob/master/.gitlab-ci.yml">.gitlab-ci.yml</a>
in that project for an example.
</p>
......@@ -53,8 +53,9 @@ title: Project Hosting
<h2 id="your-access">Your common-lisp.net access</h2>
<p>To create a new Git project, a GitLab account can be created; please
notify admin@common-lisp.net about your intent to create projects.
Due to SPAM, you need to be granted this permission manually.
notify admin@common-lisp.net about your intent to create projects,
including a fork of an existing project. Due to SPAM, you need to be
granted this permission manually.
</p>
<p>To join existing Git projects, a GitLab account can be created; with
that account, access to the existing project can be requested through
......@@ -83,10 +84,13 @@ title: Project Hosting
<h2 id="creproj">Creating a project</h2>
<p>Users can create Git projects (up to 10) themselves through the GitLab interface
in their own namespace. GitLab groups (separate namespaces) can be requested through
an e-mail to admin@common-lisp.net. GitLab groups allow hosting a project site per
group.
<p>After notifying the admins of their desire to create projects, users can
create Git projects (up to 10) themselves through the GitLab interface in
their own namespace. However, creating a GitLab group (a separate
namespace) for your new project is strongly encouraged. GitLab groups allow
nicer hosting of project web sites and better access control for related
projects worked on by multiple people. A group can be requested through an
e-mail to admin@common-lisp.net.
</p>
<p>To request a Subversion project, please contact admin@common-lisp.net. If you don't have
......@@ -130,8 +134,8 @@ title: Project Hosting
<h2 id="website">Website</h2>
<p>When the project doesn't use GitLab Pages as described above,
<tt>/project/&lt;project-name&gt;/public_html</tt> is published
from our webservers as <tt>http://common-lisp.net/project/&lt;project-name&gt;</tt></p>
<tt>/project/&lt;project-name&gt;/public_html</tt> is published from our
webservers as <tt>https://&lt;project-name&gt;.common-lisp.dev/;</tt></p>
<p>
Apache <a href="http://httpd.apache.org/docs/howto/ssi.html">Server-Side
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment