README.org 20 KB
Newer Older
1
#+TITLE: Common Lisp Project Manager - CLPM
2
#+AUTHOR: Eric Timmons
Eric Timmons's avatar
Eric Timmons committed
3
#+EMAIL: clpm-devel@common-lisp.net
4
#+OPTIONS: email:t toc:1 num:nil
5

Eric Timmons's avatar
Eric Timmons committed
6
7

*WARNING*: This software is BETA quality. I use it as my daily driver, but it
Eric Timmons's avatar
Eric Timmons committed
8
9
is still a little rough around the edges and it may accidentally eat your
files.
10

11
12
13
14
*NOTE*: This README is kept up to date on CLPM development (including
unreleased versions). For information on released versions, go to the
appropriate tag or see [[http://www.clpm.dev]].

15
16
* Description

17
18
19
20
  CLPM is a project manager for Common Lisp. It can manage projects in both
  global and project specific contexts. It would be called a "package manager"
  in most other contexts, but the use of "package" unfortunately collides with
  the use of "package" in the Common Lisp spec as a namespace for symbols.
21

Eric Timmons's avatar
Eric Timmons committed
22
  It consists of two major pieces. First is a standalone program that is
Eric Timmons's avatar
Eric Timmons committed
23
24
25
26
27
28
29
30
31
32
  responsible for all the heavy lifting of downloading and unpacking releases
  in the correct place, resolving dependencies, and managing project specific
  environments. This piece is generally referred to as CLPM, the CLPM core, or
  =clpm= and is distributed as a precompiled executable using SBCL (but it is
  possible to compile it from source). The second is a small client library
  written in portable Common Lisp that interfaces with ASDF and calls CLPM as
  necessary to find and install missing systems. This piece is generally
  referred to as the CLPM client or =clpm-client= and is meant to be loaded
  when you are interactively developing code.

Eric Timmons's avatar
Eric Timmons committed
33
34
35
36
37
38
39
40
41
42
  CLPM is licensed under the two clause BSD license (BSD-2). So, basically do
  what you want with it (while providing appropriate attribution) and don't
  complain if it breaks things. CLPM is currently beta quality so expect some
  breakage and incompatibilites going forward.

  To receive help or report issues please send email to
  [[mailto:clpm-devel@common-lisp.net][clpm-devel@common-lisp.net]]. Additionally, feel free to join the email list at
  [[https://mailman.common-lisp.net/listinfo/clpm-devel]]. To join in the
  development of CLPM, you can find the project hosted on the common-lisp.net
  Gitlab server at [[https://gitlab.common-lisp.net/clpm/clpm]].
Eric Timmons's avatar
Eric Timmons committed
43

Eric Timmons's avatar
Eric Timmons committed
44
45
* Tutorial

Eric Timmons's avatar
Eric Timmons committed
46
  It is highly recommended that you page through the [[file:tutorial/tutorial.org][tutorial]]. It will walk you
47
48
49
50
  through installing, configuring, and using CLPM. Below, you'll find a high-
  level depiction of lisp dependency management and how CLPM aids in it. See
  also the introduction to dependency management and the in-depth technical
  discussions listed at the bottom of this document.
Eric Timmons's avatar
Eric Timmons committed
51

Eric Timmons's avatar
Eric Timmons committed
52
* Installing
Eric Timmons's avatar
Eric Timmons committed
53

Eric Timmons's avatar
Eric Timmons committed
54
55
56
  CLPM is distributed in both source and binary form. For either version, first
  install the dependencies:

57
58
59
60
  * A Lisp implementation :: SBCL is currently required if you are installing
    from source. SBCL or CCL are the most tested implementations for groveling
    dependencies from .asd files (necessary if you are installing a development
    version of a project).
Eric Timmons's avatar
Eric Timmons committed
61
62
63
64
  * git :: If you want to use development versions of your dependencies.

** Prebuilt binaries
   To install CLPM in binary form, download the appropriate file from
65
66
   [[https://files.clpm.dev/clpm/][https://files.clpm.dev/clpm/]]. The current release files for each platform
   are:
67

Eric Timmons's avatar
v0.4.1    
Eric Timmons committed
68
69
   + [[https://files.clpm.dev/clpm/clpm-0.4.1-darwin-amd64.tar.gz][clpm-0.4.1-darwin-amd64.tar.gz]] :: CLPM compiled for 64bit Mac OS.
   + [[https://files.clpm.dev/clpm/clpm-0.4.1-darwin-arm64.tar.gz][clpm-0.4.1-darwin-arm64.tar.gz]] :: CLPM compiled for 64bit Mac OS with
Eric Timmons's avatar
Eric Timmons committed
70
     M1 processors.
Eric Timmons's avatar
v0.4.1    
Eric Timmons committed
71
72
   + [[https://files.clpm.dev/clpm/clpm-0.4.1-linux-amd64.tar.gz][clpm-0.4.1-linux-amd64.tar.gz]] :: CLPM compiled for 64bit Linux.
   + [[https://files.clpm.dev/clpm/clpm-0.4.1-linux-arm64.tar.gz][clpm-0.4.1-linux-arm64.tar.gz]] :: CLPM compiled for 64bit Linux
73
     on ARM processors.
Eric Timmons's avatar
v0.4.1    
Eric Timmons committed
74
75
76
   + [[https://files.clpm.dev/clpm/clpm-0.4.1-windows-amd64.msi][clpm-0.4.1-windows-amd64.msi]] :: CLPM installer for 64 bit Windows.
   + [[https://files.clpm.dev/clpm/clpm-0.4.1.tar.gz][clpm-0.4.1.tar.gz]] :: Source code, including git submodules.
   + [[https://files.clpm.dev/clpm/clpm-0.4.1.DIGESTS][clpm-v0.4.1.DIGESTS]] :: Text file containing the SHA512 sums for
Eric Timmons's avatar
Eric Timmons committed
77
     every previously mentioned file.
Eric Timmons's avatar
v0.4.1    
Eric Timmons committed
78
   + [[https://files.clpm.dev/clpm/clpm-0.4.1.DIGESTS.asc][clpm-0.4.1.DIGESTS.asc]] :: Same as =clpm-v0.4.1.DIGESTS=,
Eric Timmons's avatar
Eric Timmons committed
79
     but signed with GPG key =0x10327DE761AB977333B1AD7629932AC49F3044CE=.
Eric Timmons's avatar
Eric Timmons committed
80

81
*** GNU/Linux and Darwin
Eric Timmons's avatar
Eric Timmons committed
82
83
84
85

    After downloading the tarball and validating the SHA512 sum, unpack it and
    run =sh install.sh=. By default, it is installed to =/usr/local/=, but that
    prefix con be changed using the =INSTALL_ROOT= environment
Eric Timmons's avatar
Eric Timmons committed
86
    variable.
Eric Timmons's avatar
Eric Timmons committed
87
88
89
90
91
92
93
94
95

*** Windows

    After downloading the installer and validating the SHA512 sum, simply run
    the installer and follow the directions.

** Source Install

   See [[file:INSTALL.org][INSTALL]] for more details on installing from source.
Eric Timmons's avatar
Eric Timmons committed
96
97
98

* Quickstart

Eric Timmons's avatar
Eric Timmons committed
99
  Now that you have CLPM installed, this section walks you through how to set
100
  it up, using the packages provided by the primary Quicklisp distribution.
Eric Timmons's avatar
Eric Timmons committed
101

102
103
  First, configure CLPM to use the primary Quicklisp distribution as a source
  for packages. Create a file called =~/.config/clpm/sources.conf= (or
104
105
  =%LOCALAPPDATA%\config\clpm\sources.conf= on Windows) with the following
  contents:
Eric Timmons's avatar
Eric Timmons committed
106
107

  #+begin_src common-lisp
Eric Timmons's avatar
Eric Timmons committed
108
    ("quicklisp"
109
110
     :type :quicklisp
     :url "https://beta.quicklisp.org/dist/quicklisp.txt")
Eric Timmons's avatar
Eric Timmons committed
111
112
  #+end_src

Eric Timmons's avatar
Eric Timmons committed
113
  See [[file:docs/sources.org][sources]] for more details on configuring sources.
Eric Timmons's avatar
Eric Timmons committed
114

115
116
117
118
119
120
121
122
123
124
  Then, configure ASDF to find the CLPM client. Assuming you haven't modified
  your [[https://common-lisp.net/project/asdf/asdf.html#Controlling-where-ASDF-searches-for-systems][ASDF source registry]] too much, place the output of the following command
  at =~/.config/common-lisp/source-registry.conf.d/20-clpm-client.conf=. If
  you've modified your source registry a lot, you probably know what to do with
  the output.

  #+begin_src shell
    clpm client source-registry.d
  #+end_src

Eric Timmons's avatar
Eric Timmons committed
125
126
  Next, configure your favorite Lisp to load the client by placing the output
  of the following command in your Lisp's init file (such as =~/.sbclrc=).
Eric Timmons's avatar
Eric Timmons committed
127
128

  #+begin_src shell
129
    clpm client rc
Eric Timmons's avatar
Eric Timmons committed
130
131
  #+end_src

132
133
  Now you can start your favorite Common Lisp implementation and enter into the
  default CLPM context.
134
135
136
137
138
139
140

  #+begin_src common-lisp
    (clpm-client:activate-context "default" :activate-asdf-integration t)
  #+end_src

  When using Quicklisp metadata directly, you need to sync in order to get all
  the metadata locally (other types of sources are able to lazily sync). This
141
  step will take a while the first time as it downloads and processes every
142
143
144
145
146
147
148
  version of the distribution.

  #+begin_src common-lisp
    (clpm-client:sync :sources "quicklisp")
  #+end_src

  Now you can try loading a system, such as alexandria:
Eric Timmons's avatar
Eric Timmons committed
149

Eric Timmons's avatar
Eric Timmons committed
150
151
  #+begin_src common-lisp
    (asdf:load-system :alexandria)
Eric Timmons's avatar
Eric Timmons committed
152
153
  #+end_src

154
155
  CLPM will see that alexandria is not present locally and ask you if you would
  like to install it automatically.
Eric Timmons's avatar
Eric Timmons committed
156

Eric Timmons's avatar
Eric Timmons committed
157
158
* Project Goals

159
  CLPM is far from the only project manager available for Common Lisp[fn:1], but
Eric Timmons's avatar
Eric Timmons committed
160
161
162
  it makes very different assumptions and design choices than the other
  available solutions. In this section we describe our high level goals and how
  they affect our design decisions.
Eric Timmons's avatar
Eric Timmons committed
163

Eric Timmons's avatar
Eric Timmons committed
164
165
166
167
168
169
170
171
172
173
** Use existing libraries where possible

   If you look at Quicklisp, you'll quickly notice that the client does not
   rely on any third party code and all functionality, from an HTTP client to
   untaring, are implemented completely as part of the Quicklisp client
   project. The upsides of this strategy are that only required functionality
   is loaded into the development image, it prevents version conflicts between
   code the Quicklisp client depends on and the code you're developing depends
   on, and it helps ensure that Quicklisp works on a broad set of Common Lisp
   implementations. However, it has a major downside: the maintenance and
174
175
   development costs of the project manager are high, potentially making it
   difficult to implement new features and the project manager does not really
Eric Timmons's avatar
Eric Timmons committed
176
177
178
   drive the quality of code in the community at large higher.

   The desire to use existing libraries drives the decision for the CLPM core
179
180
181
182
183
   and client to be separated. The client has no external dependencies outside
   of ASDF/UIOP, which allows it to share all the benefits of Quicklisp's
   model, and the core never needs to be loaded into a development image, so it
   can leverage any dependency that makes sense without impacting the
   development image in the slightest.
Eric Timmons's avatar
Eric Timmons committed
184
185
186
187

** Support HTTPS

   HTTPS is becoming more and more ubiquitous. Some websites (such as Github
Eric Timmons's avatar
Eric Timmons committed
188
   and Gitlab) are only served over HTTPS and some people insist on HTTPS
Eric Timmons's avatar
Eric Timmons committed
189
190
191
   everywhere possible. This trend is not going away, therefore CLPM should
   natively support HTTPS.

Eric Timmons's avatar
Eric Timmons committed
192
193
194
195
196
197
   As LispWorks is the only Common Lisp implementation I am aware of that has
   native support for TLS, this means that CLPM has to use third party tools to
   achieve this support. This further drives the separation of the core and
   client, as CLPM can use foreign libraries to provide TLS support and this is
   not something that should be brought into a development image that does not
   otherwise need it.
Eric Timmons's avatar
Eric Timmons committed
198
199
200
201
202
203
204
205
206
207

   Additionally, Quicklisp packages are served over HTTPS. While the Quicklisp
   client cannot take advantage of that (without
   [[https://github.com/snmsts/quicklisp-https]]), CLPM can, providing a little
   more of a guarantee that packages have not been tampered with.

** Minimize footprint in development images and deployments

   Many Common Lisp implementations allow you to deliver programs by dumping an
   image to file. For most programs generated this way there is no need to have
208
209
   a bundled project manager. Therefore, there should be an easy way to use a
   project manager without leaving traces if it in your deployed system.
Eric Timmons's avatar
Eric Timmons committed
210
211
212
213

   To this end, CLPM can be used without ever loading the client (for example,
   =clpm bundle exec= configures ASDF entirely through environment variables)
   and if you do choose to use the client for better interactive development,
214
   the client is able to remove itself from the image when it is dumped.
Eric Timmons's avatar
Eric Timmons committed
215
216
217
218
219
220
221
222
223
224
225
226
227
228

** Support CI/CD workflows

   CI and CD are nice to have and nicer still when the jobs can run quickly and
   give fast feedback. To that end, CLPM is distributed in both binary and
   source form. Source for hackers or people who want to use a different
   feature set and binary for quick and easy installation in other cases.

** Support installing multiple package versions

   Ideally, updates to packages would never introduce regressions. However, we
   live in reality and this happens frequently (just look at the packages that
   get removed from Quicklisp in any given release for failing to
   build). Additionally, sometimes you just really need to use an old version
Eric Timmons's avatar
Eric Timmons committed
229
   of a dependency for some project while using the latest version of the same
Eric Timmons's avatar
Eric Timmons committed
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
   dependency in another project.

   To this end, CLPM supports installing multiple versions of the same package
   simultaneously. This is additionally an enabling feature for managing
   project specific contexts as well as global contexts.

** Support and encourage explicitly versioned systems

   ASDF allows developers to provide version numbers for their systems and
   associate version requirements with dependencies. IMHO this is a criminally
   underutilized feature of ASDF and it should be *required* that any release
   of a package in any package index should declare a version number. However,
   it is a lot of work to convince others to believe the same way and even if a
   critical mass did buy in, things wouldn't change overnight.

   Therefore, CLPM supports both the status quo (a Quicklisp package index
246
   versioned by date) as well as a [[https://gitlab.common-lisp.net/clpm/clpi][work in progress package index]] (working name
247
   of CLPI). This new Common Lisp Project Index includes both extra metadata
Eric Timmons's avatar
Eric Timmons committed
248
   about projects (such as the upstream repo which can be used to check out
249
250
   development versions) as well as information on the ASDF system version
   numbers.
Eric Timmons's avatar
Eric Timmons committed
251

252
* Introduction to Dependency Management for Common Lisp Novices
253

254
255
256
A project in Common Lisp (CL) is made of systems. The project and its systems
may have external dependencies. Using a dependency can be as easy as loading
another ~.lisp~ file from your filesystem. Or it can be much more complex,
257
258
especially when you're facing the realities of distributing useful software with
complex functionality across platforms and architectures. What's needed in that
259
260
case is a build tool. For CL, that tool is [[https://common-lisp.net/project/asdf/][ASDF]]. ASDF manages the tasks of
loading and building systems in SBCL. The ASDF documentation sums it up as:
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275

#+begin_quote
...it orchestrates the compilation and dependency management, handles some of
the portability issues, dynamically finds and loads code, and offers some
portable system access library...
#+end_quote

Systems are defined in ~.asd~ files. A single project may have 1+ ~.asd~
file(s) based on how the authors see fit to organize and/or test the components
they've written.

ASDF is also very clear in its documentation that it "will not download missing
software components for you." For that, users must turn to other tools. Manually
installing dependencies at a path ASDF recognizes is always an option.
Otherwise, [[http://quicklisp.org/][Quicklisp]] is a popular system installer with an associated online
276
277
278
registry of metadata it uses to search and retrieve systems. For CL, it's the /de facto/
choice. That being said, it falls short on modern features - no support for
HTTPS, no support for git repos, and no versioning. That's where [[https://clpm.dev][CLPM]] comes in.
279
280
281
282
283
284
285
286
287
288
289
290

CLPM's core functionality is the same as Quicklisp - (1) it tells ASDF where to
find external systems, and (2) it takes care of installing (downloading)
external systems. Let's break down what happens in each phase at a high-level.

1. CLPM registers a source registry with ASDF by calling
   ~(asdf:initialize-source-registry)~ with a list of paths to all the ~.asd~
   files a project requires
2. CLPM downloads all the systems to ~$CLPM_DATA_DIR~, which defaults to
   ~$XDG_DATA_HOME/clpm~. It may use Quicklisp's existing metadata registry or a
   newer standard called [[https://gitlab.common-lisp.net/clpm/clpi][CLPI]] to find systems

291
292
Thus, when a user asks ASDF to find an external system, ASDF knows to look in
~$CLPM_DATA_DIR~ to find the ~.asd~ and ~.lisp~ files it needs.
293
294
295
296
297
298
299
300
301
302
303
304

The last piece of core functionality of CLPM worth understanding is its ability
for users to organize dependencies using contexts. When you want to use CLPM,
you must first activate a context. Contexts may be global (usually called the
~"default"~ context) or per-project. Global contexts might be useful for
exploring systems, while per-project contexts provide a technique for locking
dependency versions for reproducible builds. Each project may also declare its
dependencies to CLPM in an aptly named ~clpmfile~.

While the concept of per-project contexts may sound similar to, say, using
~bundler~ with Ruby, ~npm~'s ~node_modules~ directory for JavaScript, or
activating a virtualenv in Python, you should be aware of how the implementation
305
details of a context and their interaction with CL impacts their behavior. For
306
307
308
309
310
311
312
313
314
starters, all systems, whether global or per-project, are installed to the same
~$CLPM_DATA_DIR~. CLPM will happily keep different versions of the same project
installed for you, though. CLPM is also recursive in that it installs
dependencies of dependencies. However, it achieves recursiveness by looking at
Quicklisp's metadata or by parsing the ~.asd~ files of dependencies, not by
parsing each dependency's ~clpmfile~. Like a ~package-lock.json~ file in a
Node.js project, CLPM writes a ~clpmfile.lock~ automatically with the full tree
of all recursive dependencies. Finally, and most importantly, it is not safe to
switch contexts on the fly. The issue with switching contexts gets to the heart
315
of how CL is different than many other languages.
316

317
Remember that a CL image (the functions and variables loaded in a REPL) is
318
319
320
321
322
323
both longer-living and more dynamic than the runtimes for other languages. You
switch contexts /from within the runtime itself/, not through an external
process (like a shell) that invokes the runtime. Thus, if you load a system with
ASDF and then switch contexts, /that first system is still loaded/. If you try
to then load another version of the same system in your new context, you're
still in the same runtime and you may find that the old system is still there.
324
In the interaction between CLPM, ASDF, SLIME, and CL, both ASDF and CLPM have
325
326
327
328
the concept of a version (more on this below). However, ASDF does not have the
concept of installing multiple versions simultaneously and switching between
them given project requirements. To ASDF, asking it to load a different version
of a system is the same as asking it to load the same system twice. The new
329
330
version might clean up the old one, but it likely won't. It depends on whether
the author(s) of the system programmed clean up behavior.
331
332
333
334
335
336
337
338

So here is what you should be thinking when you fire up SLIME:

1. I need to activate a CLPM context so CLPM knows where I'm working
2. If I haven't installed my dependencies yet, let me tell CLPM to start
   installing the external systems I need
3. Now that I have my dependencies on my machine, I can start asking ASDF to
   load them so I can use them
339
4. Time to ~C-c C-c~ on some CL!
340
341
342
343
344
345
346
347
348
349
350

*** Dos and Don'ts of Versioning with CLPM and ASDF

As noted, both CLPM and ASDF have the concept of a system version. It's best
practice to *only specify dependency versions in* ~.asd~ *files*.

Think of a ~clpmfile~ as a /constraint/ on the environment moreso than a
/definition/ of the environment. The primary function of a ~clpmfile~ is to tell
CLPM where it can find dependencies, eg. private git repositories, Quicklisp
metadata, and CLPI.

351
352
353
As a rule of thumb, you should avoid including versions in a ~clpmfile~.
However, here are a few reasons why you might want to constrain the versions of
your dependencies with CLPM.
354
355
356
357
358

1. DO version with CLPM if you want to use an unreleased version of a system.
   Say you want to use the latest commits from another repository that haven't
   made it to Quicklisp. CLPM allows you to provide a directive with a git repo
   and commit. Note however this should be considered a temporary measure and
Cameron Pittman's avatar
Cameron Pittman committed
359
   reverted to a normal release (ie. pointing to Quicklisp metadata) once the
360
   functionality you need in the system is available normally.
361
2. DO put bounds on versions with CLPM. If you want to bound the versions of
362
363
   systems for whatever reason, use CLPM.
3. DO add development-only dependencies with CLPM. This is similar to the
Cameron Pittman's avatar
Cameron Pittman committed
364
   concept of ~devDependencies~ in a JavaScript ~package.json~ file. If there
365
   are systems you need to enable unit testing or build tasks that are only needed
Cameron Pittman's avatar
Cameron Pittman committed
366
   during developing, eg. building documentation sites, CLPM is a great choice.
367
368
369

Here are a few reasons to avoid versioning with CLPM.

370
371
372
1. DO NOT add constraints to a ~clpmfile~ if you have already specified a
   version in ASDF and you are happy with the version of the system that CLPM
   installs.
373
374
375
2. DO NOT version with CLPM just because you can provide a version. Assume ASDF
   is the source of truth for the versions of dependencies.

Eric Timmons's avatar
Eric Timmons committed
376
377
378
379
* In-depth Documentation

  For more documentation on CLPM, you may find the following files useful:

Eric Timmons's avatar
Eric Timmons committed
380
381
382
383
  + [[file:docs/client.org][client]] :: Summary of CLPM's client.
  + [[file:docs/config.org][config]] :: Summary of all of CLPM's configuration options.
  + [[file:docs/sources.org][sources]] :: Summary of all supported software repositories.
  + [[file:docs/bundle.org][bundle]] :: Information on how to use CLPM to manage and repeatably
Eric Timmons's avatar
Eric Timmons committed
384
    install dependencies for a single project.
Eric Timmons's avatar
Eric Timmons committed
385
  + [[file:docs/storage.org][storage]] :: Information on where CLPM writes data to your hard drive.
Eric Timmons's avatar
Eric Timmons committed
386

Eric Timmons's avatar
Eric Timmons committed
387
388
* Footnotes

389
[fn:1] See, for example: [[https://www.quicklisp.org/beta/][Quicklisp]], [[https://github.com/fukamachi/qlot/][Qlot]], and [[https://github.com/CodyReichert/qi][Qi]].