Commit 601faff0 authored by Eric Timmons's avatar Eric Timmons
Browse files

Documentation updates

parent 4acc4776
Pipeline #834 passed with stage
in 9 minutes and 48 seconds
#+TITLE: Common Lisp Package Manager - CLPM
#+AUTHOR: Eric Timmons
#+EMAIL: etimmons@mit.edu
#+AUTHOR: CLPM Developers
#+EMAIL: clpm-devel@common-lisp.net
#+OPTIONS: email:t toc:2 num:nil
**WARNING**: This is ALPHA quality. It is being published right now so I can
start testing it and hammering at it in my normal work flow. There are probably
bugs and it may eat your files.
**WARNING**: This software is ALPHA quality. I use it as my daily driver, but it
is still a little rough around the edges and it may accidentally eat your files.
* Description
CLPM is a package manager for Common Lisp. It is focused on managing packages
for both (user) global contexts and project specific contexts.
for both user contexts and project specific contexts (bundles).
It consists of two major pieces. First is a CLI program that is responsible
for all the heavy lifting of fetching and unpacking releases in the correct
place, finding systems, and managing project specific environments. This piece
is generally referred to as the CLPM core. The second is a small client
library written in portable Common Lisp that interfaces with ASDF and calls
the CLI program as necessary to find and install missing systems. This piece
is generally referred to as the CLPM client.
Currently, the project specific context management is the most mature. Version
0.1.0 will be released when both the global and project specific features are
usable.
* Installing
CLPM requires the following programs to be installed:
+ curl :: Needed for fetching files.
+ git :: Needed to check out git repositories in bundles.
+ tar :: Needed to unpack tar files.
The easiest way to install CLPM is to download an executable
image. Instructions on that will be forthcoming once I have hosting of those
images figured out.
The next easiest way to install CLPM is to install SBCL, download the CLPM
sources (including the git submodules!), and execute the [[file:scripts/clpm-live][clpm-live]] script.
It consists of two major pieces. First is a standalone program that is
responsible for all the heavy lifting of fetching and unpacking releases in
the correct place and managing project specific environments. This piece is
generally referred to as CLPM, the CLPM core, or =clpm=. The second is a small
client library written in portable Common Lisp that interfaces with ASDF and
calls CLPM as necessary to find and install missing systems. This piece is
generally referred to as the CLPM client or =clpm-client=.
* Project Goals
CLPM is far from the only package manager available for Common Lisp[fn:1], but
it makes very different assumptions and design choices than the other
available solutions. We try to describe those choices and our high level goals
in this section.
+ Minimize footprint in client image :: It's unnecessary to load a large
amount of code into an image for a task that is only a small part of the
coding experience. This drives CLPM's separation into the core and
client. Additionally, many Lisps (I'm looking at you, SBCL) have a large
memory footprint and this helps keep that size down.
+ Use existing libraries where possible :: Because CLPM core is never loaded
into a client image, there are no issues with version conflicts or
incompatibilities between third party libraries used by CLPM and the
client image. This means that existing libraries can be leveraged where
it makes sense and we don't need to reinvent the wheel.
+ Support cleanup on image dump :: There is no need to have a package manager
in most programs. CLPM provides an option to remove the client from the
image when it is dumped (through UIOP provided functions).
available solutions. We try to describe our high level goals and how they
affect our design decisions in this section.
+ Minimize footprint in development Lisp image :: Ultimately, dependency
management should not be a large part of developing a library or
application - you should be focused on creating your cool functionality
instead of downloading and installing dependencies. So why should all the
(nontrivial amounts of) code necessary for downloading libraries, resolving
their dependencies, unpacking them, and installing them ever be present in
your development Lisp image? This drives CLPM's separation into the core and
client. Additionally, some Lisps (I'm looking at you, SBCL) have a large
memory footprint and this helps keep that down.
+ Use existing libraries where possible :: Because the CLPM core never needs
to be loaded into a development image, there are no issues with version
conflicts or incompatibilities between third party libraries used by CLPM
and the code being developed. This means that existing libraries can be
leveraged where it makes sense and we don't need to reinvent the wheel.
+ Support cleanup on image dump :: Many Common Lisp implementations allow you
to deliver programs by dumping an in memory image to file. For most programs
generated there is no need to have a bundled package manager. Therefore,
CLPM provides an option to remove the client from the image when it is
dumped (using UIOP provided functions).
+ Support multiple workflows and configurations :: We want it to be equally
easy to hack on CLPM, to use it for CI, and to use it for development by
coders uninterested in hacking on CLPM itself. This means we strive to
support both distribution of a (mostly) static program as well as using
CLPM directly from source. It also means we strive to provide reasonable
defaults that work (most) everywhere.
easy to hack on CLPM, to use it for CI, and to use it for development by
coders uninterested in hacking on CLPM itself. This means we strive to
support both distribution of a (mostly) statically linked program as well as
using CLPM directly from source.
+ Support installing multiple package versions :: CLPM supports installing
multiple versions of the same package simultaneously. This is an enabling
feature for managing project specific dependencies as well as global
dependencies.
multiple versions of the same package simultaneously. This is an enabling
feature for managing project specific dependencies as well as global
dependencies.
+ Support multiple package sources :: At first, CLPM will support only
Quicklisp-style distributions as a package source/repository. But we
envision adding support for new sources, such as Qi and a new work in
progress repository geared toward projects that follow versioning
guidelines.
Quicklisp-style distributions as a package source/repository. But we
envision adding support for new sources, such as a new work in progress
repository geared toward projects that maintain their own release schedule.
* Documentation
* Installing
For more documentation on CLPM, start at [[file:doc/overview.org][overview.org]].
CLPM is distributed in both source and binary form. Additionally, the binaries
are distributed in both dynamic and statically linked forms (note: the
binaries are built with SBCL which does not support statically linking against
the C library[fn:3], so you need to get the one that matches your OS's libc
(most likely the =gnu= variant)).
For either version, first install the dependencies:
+ tar :: Required[fn:2].
+ git :: If using bundles with git repos specified.
+ sbcl or ccl :: If systems need to be groveled for dependencies (likely
needed if you use bundles).
+ openssl :: Needed if accessing sources over https *and* compiling from
source or using the dynamically linked version.
+ sqlite :: Needed if compiling from source or using the dynamically linked
version.
To install CLPM in binary form, download the appropriate file from
[[https://common-lisp.net/project/clpm/downloads/][https://common-lisp.net/project/clpm/downloads/]]. It is highly recommended that
you use the static variants. Each release of CLPM consists of the following
files:
+ =clpm-$VERSION-x86_64-linux-gnu-dynamic= :: CLPM compiled for 64bit
Linux. Dynamically links against glibc (GNU's libc) and all supporting
libraries.
+ =clpm-$VERSION-x86_64-linux-gnu-static= :: CLPM compiled for 64bit
Linux. Dynamically links against glibc (GNU's libc) and statically linked
against all supporting libraries.
+ =clpm-$VERSION-x86_64-linux-musl-dynamic= :: CLPM compiled for 64bit
Linux. Dynamically links against musl libc and all supporting libraries.
+ =clpm-$VERSION-x86_64-linux-musl-static= :: CLPM compiled for 64bit
Linux. Dynamically links against musl libc and statically linked against all
supporting libraries.
+ =clpm-$VERSION-x86_64-windows-dynamic= :: CLPM compiled for 64bit
Windows. Dynamically links against all supporting libraries.
+ =clpm-$VERSION-x86_64-windows-static= :: CLPM compiled for 64bit
Windows. Statically linked against all supporting libraries.
+ =clpm-$VERSION.DIGESTS= :: Text file containing the SHA512 sums for every
previously mentioned file.
+ =clpm-$VERSION.DIGESTS.asc= :: Same as =clpm-$VERSION.DIGESTS=, but signed
with GPG key =0x10327DE761AB977333B1AD7629932AC49F3044CE=.
After downloading the binary and validating the SHA512 sum, make sure the
file is executable and save it as =clpm= somewhere on your PATH.
The next easiest way to install CLPM is to install SBCL, clone the CLPM
sources (including the git submodules!), and link the [[file:scripts/clpm-live][clpm-live]] script as
=clpm= somewhere on your PATH. Alternatively, you can build a dynamically or
statically linked executable by running one of the following:
#+begin_src shell
sbcl --script scripts/clpm-build-dynamic-sbcl
sbcl --script scripts/clpm-build-static-sbcl
#+end_src
After ensuring =clpm= is on your path you can install the client library and
(optionally) add it to your ASDF search path by running:
#+begin_src shell
clpm client
#+end_src
* Quickstart
Now that you have CLPM installed, you can begin using it! First, you need to
add a software repository. You likely want to start with the main Quicklisp
distribution, so create a file called =~/.config/common-lisp/clpm/clpm.conf=
with the following contents:
#+begin_src common-lisp
(clpm-config
:sources
("quicklisp"
(:type :quicklisp
:url "https://beta.quicklisp.org/dist/quicklisp.txt"
:force-https t)))
#+end_src
This configures CLPM to use the primary Quicklisp distribution and configures
it to always fetch files from it over HTTPS [fn:5] [fn:6]. See [[file:doc/config.org][config.org]] for
more details.
Then, sync your local copy of the Quicklisp metadata by running:
#+begin_src shell
clpm sync
#+end_src
Syncing may take a while the first time. When it's finished, you can install a
project, such as CFFI, by running:
#+begin_src shell
clpm install cffi
#+end_src
This will install all ASDF systems belonging to the CFFI project (currently
=cffi=, =cffi-uffi-compat=, =cffi-toolchain=, =cffi-tests=, =cffi-libffi=,
=cffi-grovel=, and =cffi-examples=) and all their dependencies.
If you want to install only a single system and its dependencies[fn:4] you can use
the =-s= (system) option, such as:
#+begin_src shell
clpm install -s cffi
#+end_src
You can test this was successful by starting your favorite Lisp and running:
#+begin_src common-lisp
(require :asdf)
(asdf:load-system :cffi)
#+end_src
Note how you do not need to load any CLPM code inside your Lisp image in order
to load CFFI after installing it. (Assuming you have ASDF 3+ installed).
* In-depth Documentation
For more documentation on CLPM, you may find the following files useful:
+ [[file:doc/config.org][config.org]] :: Summary of all of CLPM's configuration options.
+ [[file:doc/sources.org][sources.org]] :: Summary of all supported software repositories.
+ [[file:doc/bundle.org][bundle.org]] :: Information on how to use CLPM to manage and repeatably
install dependencies for a single project.
* Footnotes
[fn:6] On Windows, CLPM will fetch over HTTPS, but it will *not* currently
validate the certificates. See [[https://gitlab.common-lisp.net/clpm/clpm/issues/1][issue#1]] for more info.
[fn:5] All files in the primary Quicklisp distribution are served over both
HTTPS and HTTP, even though the Quicklisp client cannot use HTTPS itself.
[fn:4] Really, the entire project the system belongs to will be installed, but
only the dependencies of the specified system will be installed.
[fn:3] If you know a way to statically link SBCL against libc, please let me
know!
[fn:1] See, for example: [[https://www.quicklisp.org/beta/][Quicklisp]], [[https://github.com/fukamachi/qlot/][Qlot]], and [[https://github.com/CodyReichert/qi][Qi]].
[fn:2] Using the archive library for a full Common Lisp solution is on the
roadmap, but it needs a decent amount of work.
#+TITLE: CLPM Bundle
#+AUTHOR: Eric Timmons
#+EMAIL: etimmons@mit.edu
#+AUTHOR: CLPM Developers
#+EMAIL: clpm-devel@common-lisp.net
This file describes the "bundle" capabilities of CLPM. Bundles are meant to ease
the problems of repeatable deployment and testing in the CL world - i.e.,
......@@ -14,19 +14,19 @@ repos simultaneously!).
A CLPM bundle is a collection of systems locked at specific versions. A bundle
is fully defined by a =clpmfile.lock= file, which is itself created from a
=clpmfile= file. Once a lock file is generated, the bundle commands allow you to
perform tasks such as packaging the whole bundle for easy deployment/sharing and
running Lisp processes that can load systems in the bundle with zero
configuration or helpers needed (other than ASDF, of course).
perform tasks such as running Lisp processes that can load systems in the bundle
with zero configuration or helpers needed (other than ASDF, of course).
* Files
** clpmfile
A =clpmfile= describes what projects or systems a user wants to place in a
bundle and any constraints she has on the versions of those objects. It is a
file that consists of a series of directives where each directive is a list
starting with a keyword. The directives are described below. Additionally, the
first directive in the file must be the ~:api-version~ directive. This
document describes version ="0.1"=.
file, typically located at the root of a source code repository, that
consists of a series of directives where each directive is a list starting
with a keyword. The directives are described below. Additionally, the first
directive in the file must be the ~:api-version~ directive. This document
describes version ="0.1"=.
*** ~:api-version~
......@@ -103,10 +103,11 @@ configuration or helpers needed (other than ASDF, of course).
* Configuration
Every bundle command reads the file =.clpm/bundle.conf= (if it exists) and
merges the configuration defined in that file into CLPM's config. Currently,
all configuration sections are merged, in a future version a whitelist of
configuration options will be defined.
Every bundle command reads the file =.clpm/bundle.conf= (if it exists,
relative to the =clpmfile.lock=) and merges the configuration defined in that
file into CLPM's central config. Currently, all configuration sections are
merged, in a future version a whitelist of configuration options will be
defined.
* Commands
** =clpm bundle install=
......@@ -132,6 +133,11 @@ configuration or helpers needed (other than ASDF, of course).
+ =CLPM_BUNDLE_CLPMFILE_LOCK= :: Set to the path to the clpmfile.lock that
defines the bundle.
** =clpm bundle update=
Update the lock file to point to the latest versions available that satisfy
the constraints in the =clpmfile=.
* Comparisons
In the Common Lisp world, CLPM's bundle is most similar to [[https://github.com/fukamachi/qlot][Qlot]]. Unlike Qlot,
......
#+TITLE: CLPM Configuration
#+AUTHOR: CLPM Developers
#+EMAIL: clpm-devel@common-lisp.net
CLPM is configured using a file named =clpm.conf=. CLPM constructs a list of
directories to search for this file and the first instance found wins. The
......@@ -20,7 +22,7 @@ The following describe the configuration options for clpm.
* ~:archives~
+ ~:tar-method~ :: A keyword specifying which tar implementation to
use. Defaults to =:auto=.
use. Defaults to =:auto=.
** ~:tar~
......@@ -35,7 +37,7 @@ The following describe the configuration options for clpm.
for PROJECT-NAME, use a local checkout instead.
+ ~:path~ :: Path to the folder containing the checkout of the git
repository to use.
repository to use.
* ~:git~
** ~:remotes~
......@@ -43,11 +45,11 @@ The following describe the configuration options for clpm.
Provide configuration for interacting with HOSTNAME to fetch git repos.
+ ~:username~ :: A string containing the username to use when connecting to
the server.
the server.
+ ~:password~ :: A string containing the password to use when connecting to
the server.
the server.
+ ~:method~ :: A keyword describing how to connect to the server. Can be one
of:
of:
+ ~:https~ :: Connect using HTTPS.
+ ~:http~ :: Connect using HTTP.
+ ~:ssh~ :: Connect using SSH.
......@@ -60,17 +62,17 @@ The following describe the configuration options for clpm.
the name HEADER-NAME.
+ ~:secure-only-p~ :: If non-NIL, send this header only on secure
connections.
connections.
+ ~:value~ :: The value to send for the header. If a string, the contents
of the string are used literally. If a pathname naming an
executable file, that file is executed and whatever it prints
to its standard output is used as the value. If a pathname
naming a non-executable file, the contents of the file are
sent.
of the string are used literally. If a pathname naming an executable
file, that file is executed and whatever it prints to its standard output
is used as the value. If a pathname naming a non-executable file, the
contents of the file are sent.
* ~:http-client~
+ ~:method~ :: The default method to use to as an HTTP client. Defaults to
=:auto=.
=:auto=.
** ~:curl~
Settings for the curl method that uses an external curl program.
......
#+TITLE: CLPM Overview
#+AUTHOR: Eric Timmons
#+EMAIL: etimmons@mit.edu
+ [[file:config.org][config.org]]
+ [[file:sources.org][sources.org]]
+ [[file:bundle.org][bundle.org]]
#+TITLE: CLPM Sources
#+AUTHOR: CLPM Developers
#+EMAIL: clpm-devel@common-lisp.net
A repository of Common Lisp projects is called a "source". The source you are
most likely familiar with is Quicklisp, but CLPM is designed to use multiple
sources and source types. This allows you to easily use CLPM for things such as
internal projects that can't be released to public repositories.
most likely familiar with is Quicklisp, but CLPM is designed to be extended to
support multiple sources and source types. This will allow you to easily use
CLPM for things such as internal projects that can't be released to public
repositories.
Currently, the only type of source that CLPM publicly supports is a Quicklisp
style source. The author is currently working on and testing a new source type
that improves upon Quicklisp, though (such as tracking of individual system
versions). If you are interested in trying out that source type, feel free to
reach out.
The type of every source must can be specified with a keyword.
The type of every source must be specified with a keyword.
* Quicklisp
......@@ -27,10 +24,10 @@ The type of every source must can be specified with a keyword.
+ =url= :: The URL to the subscription metadata file.
+ =force-https= :: If true, the protocol for all URLs is changed to https
before downloading any files. This is useful because all
URLs contained in the primary Quicklisp distribution have
http as the protocol even though they are also served over
https.
before downloading any files. This is useful because all URLs contained in
the primary Quicklisp distribution have http as the protocol even though
they are also served over https. If the =url= uses the =https= scheme,
=force-https= is implicitly set to true and cannot be disabled.
The following table can be added to your CLPM config file to use the primary
Quicklisp distribution.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment