best_practices.md 39.7 KB
Newer Older
1 2 3 4 5 6
# ASDF Best Practices

This document presents the current best practices and conventions
for using ASDF 3, as of 2017.
It is not a tutorial, though it starts like one,
because it assumes for each category of ASDF user
7
(beginner, simple user, more elaborate user)
8 9
that he already knows what seems to be common knowledge among such users,
and tries to complete this knowledge with less obvious points
10
that are often wrong in systems seen in the wild.
11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

[TOC]

## Trivial Examples

Let's start with some trivial examples.
We'll see below how these examples evolve as systems grow more complex.

### Trivial Uses of ASDF

#### Loading a System

The most trivial use of ASDF is by calling `(asdf:load-system :foobar)`
to load your library.
Then you can use it.
For instance, if it exports a function `some-fun` in its package `foobar`,
then you will be able to call it with `(foobar:some-fun ...)` or with:

    (in-package :foobar)
    (some-fun ...)

#### Testing a System

To run the tests for a system, you may use:

    (asdf:test-system :foobar)

The convention is that an error SHOULD be signalled if tests are unsuccessful.

#### Designating a System

Using keywords to name systems is good and well at the REPL.
However, when writing a program, a bootstrap script, or a system definition,
you SHOULD follow the good style of writing names in the canonical way;
45 46
and the canonical name of a system is a string, which by convention is in lower-case
(indeed any symbol used as system designator is downcased by `asdf:coerce-name`).
47 48 49 50 51 52

Thus, the proper way to designate a system in a program is with lower-case strings, as in:

    (asdf:load-system "foobar")
    (asdf:test-system "foobar")

53 54 55 56 57 58 59 60 61 62 63 64 65
(Historical note: MK-DEFSYSTEM and before it Genera's SCT
used string-upcase to normalize system names,
and use that as filenames, which made sense on Genera's filesystem
or using logical pathnames as common with MK-DEFSYSTEM.
Dan Barlow's ASDF, to "Play Nice With Unix", preferred lower-case file names,
and used string-downcase to normalize symbols into strings;
at the same time, he allowed you to specify mixed case or upper-case names by using a string.
The latter was probably a mistake, and I strongly recommend against using anything but lower-case.
If that weren't allowed, there would be a case for making lower-case symbols the default syntax,
and let the various case conversions do their job without quotes.
But that is not where we are.
In any case, system names designate `.asd` files, not Lisp bindings,
and this determines their syntax.)
66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153

### Trivial System Definition

A trivial system would have a single Lisp file called `foobar.lisp`.
That file would depend on some existing libraries,
say `alexandria` for general purpose utilities,
and `trivia` for pattern-matching.
To make this system buildable using ASDF,
you create a system definition file called `foobar.asd`,
with the following contents:

    (defsystem "foobar"
      :depends-on ("alexandria" "trivia")
      :components ((:file "foobar")))

Note how the type `lisp` of `foobar.lisp`
is implicit in the name of the file above.
As for contents of that file, they would look like this:

    (defpackage :foobar
      (:use :common-lisp :alexandria :trivia)
      (:export
       #:some-function
       #:another-function
       #:call-with-foobar
       #:with-foobar))

    (in-package :foobar)

    (defun some-function (...)
      ...)
    ...

#### Using the system you defined

Assuming your system is installed under the `~/common-lisp/` hierarchy
or some other filesystem hierarchy already configured for ASDF,
you can load it with: `(asdf:load-system "foobar")`,
or shorter `(asdf:make :foobar)`

If your Lisp was already started when you created that file,
you may have to `(asdf:clear-configuration)` to re-process the configuration.


### Trivial Testing Definition

Even the most trivial of systems needs some tests,
if only because it will have to be modified eventually,
and you want to make sure those modifications don't break client code.
Tests are also a good way to document expected behavior.

The simplest way to write tests is to have a file `foobar-tests.lisp`
and modify the above `foobar.asd` as follows:

    (defsystem "foobar"
      :depends-on ("alexandria" "trivia")
      :components ((:file "foobar"))
      :in-order-to ((test-op (test-op "foobar/tests"))))

    (defsystem "foobar/tests"
      :depends-on ("foobar" "fiveam")
      :components ((:file "foobar-tests"))
      :perform (test-op (o c) (symbol-call :fiveam '#:run! :foobar)))

The `:in-order-to` clause in the first system
allows you to use `(asdf:test-system :foobar)`
which will chain into `foobar/tests`.
The `:perform` clause in the second system does the testing itself.

In the test system, `fiveam` is the name of a popular test library,
and the content of the `perform` method is how to invoke this library
to run the test suite `:foobar`.
Obvious YMMV if you use a different library.

#### Note on ASDF 2 compatibility

The `:in-order-to ((test-op (test-op ...)))` idiom will not work with ASDF 2:
attempts to `test-system` will result in failure due to circular dependencies;
however, `load-system` will still work.
As for the `:perform` method, you need to specify `:after` right after `test-op`
for ASDF 2 to accept the `defsystem` form at all (but still not make the `:in-order-to` work);
so it's probably OK if your test system is in a separate `.asd` file,
but if you want backward compatibility and it's a secondary system, you'll use `:after`.
Making things work completely on ASDF 2 as well as ASDF 3 would take a lot of pain,
and would be colloquial for neither.

But *friends don't let friends use ASDF 2*.
ASDF 3 has been available since 2013, and
154
all implementations have now been providing it for a long time.
155
Many systems, and growing, assume ASDF 3, possibly some you depend on,
156 157
and it's pointless to maintain compatibility with a moribund system
(especially so if you don't also check that all your dependencies do, too).
158 159
Upgrade your implementations and/or at least your ASDF,
and tell your friends to do as much.
160 161
The [very founding principle of ASDF 2](http://fare.livejournal.com/149264.html) (and 3)
was to provide users with
162 163 164 165 166 167 168
the ability to install a newer ASDF on top of an old one to fix its bugs
(an ability that did not exist with ASDF 1);
thus, users do not have to maintain compatibility with antique bugs.

These days, I even recommended that you should freely rely
on features of ASDF 3.1 (2014) not present in ASDF 3.0 (2013), such as:
`~/common-lisp/` being included in the source-registry by default;
169
`run-program` being full-fledged with input and error-output support;
170
or the `package-inferred-system` feature.
171
Conversely, if you want indefinite backward compatibility, why stop at ASDF 2?
172 173 174 175 176
Why not also support ASDF 1, and MK-DEFSYSTEM, and
the original Lisp Machine DEFSYSTEM?
A good rule of thumb for me is that it is not worth writing workarounds
for bugs that were fixed or features that were changed two years ago or more.
Two years seems to be the time it takes for a release of ASDF
177
to become ubiquitously available by default on all implementations;
178 179 180
and, remember, if a user must use an older implementation,
he can always trivially install a newer ASDF on top of it.
Thus, by 2019, people should not be shy about
181
dropping support for ASDF versions older than 3.2 or 3.3.
182
And even before then, if you need a recent ASDF, just document it,
183 184
and tell your users to upgrade their implementation and/or
install a recent ASDF on top of their implementation's.
185 186 187 188 189


### Trivial Packaging

In the previous testing code, `symbol-call` is a function defined in package `uiop`.
190
It helps deal with the fact that the package `:fiveam` isn't defined yet
191 192 193 194 195 196 197 198 199
at the time the `defsystem` form is read.
Thus you can't simply write `(fiveam:run! :foobar)`, as
this might cause a failure at read-time when file `foobar.asd` is first read
and the system `fiveam` isn't loaded yet.
This of course also holds for packages defined by system `foobar` itself.

To get a symbol in it that is not a function being called,
you can also use `(find-symbol* :some-symbol :foobar)`,
and you can use `symbol-value` to get the value bound to such a symbol as denoting a variable.
200 201 202
For complex expressions, you can use `eval` and `read-from-string`,
as in `(eval (read-from-string "(foobar:some-fun foobar:*some-var*)"))`,
or equivalently, also from `uiop`: `(eval-input "(foobar:some-fun foobar:*some-var*)")`.
203 204 205

System definition files are loaded with the current `*package*` bound to the package `:asdf-user`,
that uses the packages `:cl`, `:asdf` and `:uiop`.
206
Therefore, you don't need to and you SHOULD NOT specify `cl:`, `asdf:` or `uiop:` prefixes
207
when referring to symbols in these respective packages.
208 209 210 211 212
It is considered bad form to do so, unless you changed the package,
and even then, your package should probably use these packages, too.
In particular you SHOULD NOT to write `(asdf:defsystem "foobar" ...)`
instead of `(defsystem "foobar" ...)`.
Also, you SHOULD NOT use `(in-package ...)` or `(defpackage ...)`
213 214 215 216 217 218 219 220 221 222 223 224 225
if all you do can legitimately be done in package `ASDF-USER`.

System definition files are loaded by a special function `asdf::load-asd`;
this function locally binds the current `*package*` to the `:asdf-user` package,
but it also does many other things.
You MUST NOT ever load a `.asd` file using `cl:load`,
as this will not work in general, and may fail in subtle or spectacular ways.
You MUST always use `asdf::load-asd` for that purpose
(the function is exported in ASDF 3.2 and above, but not in earlier versions).
You SHOULD NOT encourage the illusion that a `.asd` file can be loaded with `cl:load`
by using `(in-package :asdf-user)` or anything,
or by using the package prefix `asdf:` for `defsystem` and other such symbols.
You SHOULD be using the `slime-asdf` extension to SLIME
226 227
if you are going to edit `.asd` file and then load them from SLIME,
as it will automatically use `load-asd` to load the file contents.
228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313

#### Digression about symbols and packages

A crucial notion in Common Lisp is that of *symbols*, to which are associated
functions, variables, macros, properties, and other meanings.
These symbols are organized in a two-level namespace that is global to a given Lisp image.
Symbols are the second level, and the first level are *packages*.
Symbols and packages are traditionally uppercase internally,
but you traditionally write them lower-case in source code;
the Common Lisp reader is *case-converting*.
These internal representation details matter, because they are exposed
when you either write macros at compile-time or introspect the system at runtime.

The conventional way to refer to a package is with its name designated by a lower-case keyword,
starting with the colon character `:`.
For instance, the standard package `COMMON-LISP` is commonly designated as `:common-lisp`
(or `(find-package :common-lisp)` if you need an actual package object).
Keywords are themselves symbols in the magic package `KEYWORD`,
where each interned symbol is also a constant bound to its own name as a value.
Using a keyword as designator rather than a string ensures that we can maintain
the same convention of using lower-case in source code while the runtime will use uppercase.

## Simple Examples

As systems grow, the above pattern quickly becomes insufficient,
but systems can still remain simple.

### Simple Uses of a System

#### Building a System

Some systems offer operations
that are neither loading in the current image, nor testing.
Whichever operation a system is meant to be used with, you may use it with:

    (asdf:make :foobar)

This will invoke `build-op`, which in turn will depend on
the `build-operation` for the system, if defined, or `load-op` if not.
Therefore, for usual Lisp systems that want you to load them,
the above will be equivalent to `(asdf:load-system :foobar)`,
but for other Lisp systems, e.g. one that creates a shell command-line executable,
`(asdf:make ...)` will do the Right Thing™, whatever that Right Thing™ is.

#### Inspecting a System

To look at what ASDF knows of your system, try:

    (describe (asdf:find-system "foobar"))

If you're looking for the `.asd` file, try `(asdf:system-source-file "foobar")`.
For the directory in which the `.asd` file resides, try `(asdf:system-source-directory "foobar")`.
For a specific file under that directory try
`(asdf:system-relative-pathname "foobar" "path/to/the/file.extension")`.

#### Other Operations

ASDF has the concept of an *operation* that can act upon a system (or a smaller component thereof).
Typical operations that matter to end-users include:

  * `load-op` for loading a system in the current Lisp image, as used by `asdf:load-system`,
  * `test-op` for running tests associated to the system, as used by `asdf:test-system`,
  * `build-op` for doing whatever build operation is associated to the system,
    (which by default is `load-op` if the system didn't override it),
    as used by `asdf:make`.

Further operations of interest to users include:

  * `compile-op`
    (ensure the system is compiled, without necessarily loading all of it, or any bit of it),
  * `load-source-op`
    (load the system from source without compiling),
  * `compile-bundle-op`
    (create a single fasl for the entire system, for delivery),
  * `monolithic-compile-bundle-op`
    (create a single fasl for the entire system *and all its transitive dependencies*, for delivery),
  * `image-op`
    (create a development image with this system already loaded, for fast startup),
  * `program-op`
    (create a standalone application, which we will see below), etc.

Whichever operation you want, `(asdf:operate `*operation* *system*`)`
will ensure this operation is performed on that system
(after all the necessary dependencies of such an action).
A short-hand `asdf:oos` is available for `asdf:operate`
(the name `oos` is an old acronym for `operate-on-system`,
314 315
and both functions existed as synonyms in the old MK-DEFSYSTEM;
the latter function doesn't exist in ASDF, which instead has just `operate`).
316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352

The operation is typically specified as a symbol that names the operation class.
Since ASDF 3, you can also use a keyword to specify an action in the ASDF package.
Thus, `(asdf:oos :load-op :foobar)` is equivalent to `(asdf:load-system :foobar)`.

### System Naming

#### Primary Systems

ASDF has a notion of *primary system*,
that it can find in configured parts of the filesystem (the *source-registry*),
in a file that has the same name and the type `asd`.
Thus, primary system `foobar` is defined in a file `foobar.asd`.

While arbitrary strings are accepted in system names, it is strongly discouraged
to use anything but lower-case ASCII letters and digits for primary system names,
plus the separators `-` (dash) and `.` (dot).
The `.` itself is only recommended in primary system names that are part of an informal hierarchy;
for instance the popular library `iolib` contains many related systems:
`iolib.asdf`, `iolib.base`, `iolib.common-lisp`, `iolib.conf`, `iolib.examples`,
`iolib.grovel`, `iolib.tests`.
The main one is ostensibly `iolib`, but it contains many systems,
and for some reasons (notably proper phase separation)
they cannot all be secondary systems in the same file.

#### Secondary Systems

A *secondary system* is a system defined in the same file as a primary system.
By convention, its name starts the same as the file's primary system,
followed by a slash `/` and by a *suffix* made of some arbitrary characters,
preferrably ASCII letters and digits
plus the separators `-` (dash), `.` (dot) and `/` (slash).
We already saw an example of it with system `foobar/tests`
in the trivial testing definition above.

The convention above allows ASDF 3 and later to find a secondary system by name,
by first looking for the associated primary system.
353 354 355 356 357 358
ASDF 3.2 or later will issue a `warning` when you violate this naming convention.

(Historically, ASDF 1 (and 2) allowed arbitrary names for secondary systems,
and in practice many people used `foo-test` or such for secondary system names in `foo.asd`;
however, ASDF 1 (and 2) couldn't find those systems by name, and horrific bugs could happen
if a system was simultaneously defined in multiple files.)
359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431

### Simple System Definition

A simple system may be made of many files.

Typically, in `package.lisp` (or `packages.lisp`)
you'll define the package (or packages)
used by all the files in your system
in one or more forms such as `(defpackage :foobar ...)`.

Then, in a package `utils.lisp` you'll define utility macros and functions
that you're using throughout your system;
maybe some of them deserve to be moved to `alexandria` and other utility systems;
and maybe they already exist out there and you just haven't looked hard enough.

Then a file `foobar.lisp` defines the meat of your system.

Your system definition will look like:

    (defsystem "foobar"
      :depends-on ("alexandria" "trivia")
      :serial t
      :components
      ((:file "package")
       (:file "utils")
       (:file "foobar")))

The `:serial t` indicates that each of these files depends on (all) the previous.

As your system grows some more, soon enough instead of a single file `foobar.lisp`,
you will have several files, one for each aspect of your system, such as
`foo.lisp`, `bar.lisp` as well as `foobar.lisp`.
Moreover, `:serial t` soon becomes inappropriate:
it will make your code slower to compile, but also to read.
Indeed, those who read the code won't be readily able to tell
which parts of the code they need to keep active in their brains
to understand the code at hand.
Instead, you may prefer to explicitly represent the dependencies
between the components of your system using `:depends-on` clauses
as follows:

    (defsystem "foobar"
      :depends-on ("alexandria" "trivia" "trivia.ppcre")
      :components
      ((:file "package")
       (:file "utils" :depends-on ("package"))
       (:file "foo" :depends-on ("utils"))
       (:file "bar" :depends-on ("utils"))
       (:file "foobar" :depends-on ("foo" "bar"))))

Out of good style, you SHOULD still list the components in an order that makes sense,
such that the readers can read the files and mentally rebuild the system.
However, note that this order, if coherent, will be respected
only in ASDF 3.3 or later, due to a bug in earlier versions of ASDF.
But that precise order shouldn't matter, or it should be reflected
in the `:depends-on` declarations (or in a `:serial t` declaration).

### Simple Packaging

#### Initial Package for a Lisp File

You MAY assume that the current package uses `CL` at the beginning of a file,
but you MUST NOT assume that it is any particular package at this point:
right now, it *is* guaranteed to be the `CL-USER` package of the underlying implementation,
but it is conceivable that in some indeterminate future,
some extension to CL may provide a well-defined portable alternative that ASDF would use.

Therefore, the sane way to write a Lisp file is that it SHOULD start
with an `in-package` form, optionally preceded by a `defpackage` form
(or a `uiop:define-package` form).
You SHOULD NOT write `cl:in-package` or precede your `defpackage` with an `(in-package :cl-user)`
(which is stupid, because to be pedantic you'd have to `(cl:in-package :cl-user)`,
at which point you may as well `(cl:defpackage ...)` and `(cl:in-package ...)`).
432 433
If it's a regular `cl-source-file`, it can assume the language is CL indeed,
and that the readtable something reasonable, etc.
434 435 436 437 438 439 440 441 442 443 444 445

#### Using symbols from ASDF and UIOP

You MAY use any of the symbols documented and exported by ASDF or UIOP.
Actually, it is warmly recommended to use them everywhere that it matters,
instead of less documented or less portable alternatives.

You MUST NOT use `asdf:run-shell-command`, `asdf:system-definition-pathname`,
or other deprecated functions that were once recommended in the time of ASDF 1.
They will be removed in the near future (one to two year horizon).
ASDF 3.2 will issue a `style-warning` when you do, and ASDF 3.3 will issue a full `warning`,
which will break the SBCL build.
446
See `backward-interface.lisp` for a list of deprecated function — or just heed the damn warnings.
447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495

### Simple Testing

Test systems can also be divided in multiple files.
If possible (which is not always the case), the file names for test files
should match file names for regular code files,
so that the file that tests `bar.lisp` will be called `bar-test.lisp` or `bar-tests.lisp`
(pick singular or plural, but have a story for it and be consistent about it).
To keep things tidy as the test system grows,
you may even put all test files in a subdirectory `t/`, `test` or `tests/`.
Your test system definition may then look like:

    (defsystem "foobar/tests"
      :depends-on ("fiveam" "foobar")
      :pathname "t/" ;; specify the subdirectory
      :components
      ((:file "test-suite")
       (:file "utils-test" :depends-on ("test-suite"))
       (:file "foo-test" :depends-on ("test-suite"))
       (:file "bar-test" :depends-on ("test-suite"))
       (:file "foobar-test" :depends-on ("test-suite")))
      :perform (test-op (o c) (symbol-call :foobar/tests :run-test-suite)))

As the system and its test system both grow, the test system may be moved to its own file
`foobar-tests.asd` or `foobar.tests.asd` where it is its own primary system:

    (defsystem "foobar-tests" ...)

### Other secondary systems

Other secondary systems may be created beyond test systems:
for instance systems that provide independent aspects of the system,
or optional add-ons to it.
One case is a command that makes the Lisp functionality accessible from a Unix shell.

#### Delivering an Executable

To build an executable, define a system as follows
(in this case, it's a secondary system, but it could also be a primary system).
You will be able to create an executable file `foobar-command`
by evaluating `(asdf:make :foobar/executable)`:

    (defsystem "foobar/executable"
      :build-operation program-op
      :build-pathname "foobar-command" ;; shell name
      :entry-point "foobar::start-foobar" ;; thunk
      :depends-on ("foobar")
      :components ((:file "main")))

496 497 498 499 500 501 502 503 504
The `build-pathname` gives the name of the executable;
a `.exe` type will be automatically added on Windows.
As a horrible kluge that may go away in the future,
the output will be created in the system's directory (where the `foobar.asd` file resides)
when the `build-operation` matches the requested operation.
A future version of ASDF will probably instead have some `:output` argument to `operate`,
or some such thing, and drop this ugly special case.
You have been warned. Contact me if you want that sooner, or not at all.

505
There, file `main.lisp` defines a function `start-foobar` in package `foobar`,
506 507 508 509 510 511
that takes no argument, and initializes and starts the executable ---
after `uiop` calls its own `*image-restore-hook*` and evaluates any provided `*image-prelude*`.
Importantly, libraries may register functions to call in the `*image-restore-hook*`
using `register-image-restore-hook` (see also `register-image-dump-hook`);
UIOP and ASDF themselves make use of this facility.
Typically, `start-foobar` will be defined as something like:
512 513 514 515 516 517

    (defun start-foobar () (main (uiop:command-line-arguments)))

Where function `main` parses the arguments
(a list of strings, excluding the magic C `argv[0]`, which can be computed as `(uiop:argv0)`)
and does whatever its magic.
518 519
You may want to use the full-featured `net.didierverna.clon` or my small `command-line-arguments`
or one of a slew of other libraries to parse the command-line arguments.
520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551
You may want to use `cl-scripting` to nicely wrap Lisp code into execution contexts
that handle errors in a nice(r) user-visible way for the shell user.
You may want to use `inferior-shell` if your program in turn invokes other shell programs.

Instead may also use `cl-launch` to build executables, or `buildapp`.
`cl-launch` is largely compatible with ASDF
(indeed, a lot of code formerly written as part of `cl-launch`
was later made part of ASDF 3's UIOP).
`buildapp`, that came before ASDF 3 but after `cl-launch`,
has a slightly incompatible convention where a main function is called with
a list of arguments that includes the `argv[0]`.
(`cl-launch` and after it UIOP had many good enough reasons
to start the argument list at the "user arguments", if only because
when invoking a Lisp implementation via `cl-launch`,
the process' `argv[0]` may not be available or meaningful,
whereas the user arguments may be only a subset of the actual process' arguments.)
Both `cl-launch` and `buildapp` have similar functionality
to handle multicall binaries à la [Busybox](http://busybox.net/),
with the same incompatibility as above.

#### System Connections

Sometimes, a system can provide an extension to another system.
For instance, if you use both `metacopy` and `contextl`,
you may be interested in some system `metacopy/with-contextl`
that creates synergies between these two systems.

There exists a system `asdf-system-connections` that will allow you
to define such system connections that are *automatically* loaded
when both the connected systems are loaded.
While I debugged that mechanism and made sure it works with ASDF 3,
I recommend against using it, because it introduces side-effects within the build.
552
Instead I recommend explicitly loading the system connections
553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713
as part of the larger system that will use them.

## More Elaborate Examples

### More Elaborate Uses of ASDF

#### force

Sometimes, you want to force ASDF to re-build some system.
At those times, you can pass the `:force` argument to `asdf:operate`
(or its wrappers `asdf:load-system`, `asdf:test-system`, `asdf:oos`, etc.).
Passing an argument `t` will force rebuild of just the system, and none of its dependencies
(and that also means none of the dependencies that happen to be secondary systems
with the same primary system name).
Passing a list of system designators (preferrably lower-case strings)
will force the specific systems to be rebuilt (if they appear in the build plan at all, that is).
Finally, passing `:all` as argument will force a rebuild of everything, including all dependencies.

Thus, if you ran tests that use the `prove` test framework,
in which loading the files is itself the test,
and you want to force a re-run, even though ASDF might be satisfied
with already having loaded the files, then you can use:
`(asdf:load-system :clack-test :force t)`

Note that it is strongly recommended to not have any non-determinism or side-effects
that are not declared to ASDF, that would cause the forcing to be meaningful.
Forcing is thus a debugging feature for ASDF systems that fail this good practice.
In the case of `prove`, we will have to work with its author so that the correct way
to use it doesn't violate ASDF invariants, but instead properly declare that
ASDF should not consider tests already run.

#### force-not

The converse of `:force` is `:force-not`, and you can specify a list of systems to not rebuild.
In this context `t` means "everything but this system" rather than " this system".
`force-not` takes precedence over `force`, and by default includes a list of "immutable" systems
that may be used when delivering extensible applications to customers
(See function `asdf::register-immutable-system`).

Note that these flags are only for use by the user at the toplevel.
You MUST NOT call `asdf:operate` with `:force` or `:force-not` from within a build.
Actually, you should probably not explicitly use `asdf:operate` at all,
except maybe inside a `.asd` file in cases where `defsystem-depends-on` isn't sufficient.

#### Require

You SHOULD NOT use `cl:require` as a substitute for `asdf:load-system`.
You SHOULD NOT use `asdf:require-system` except at the toplevel.

Back in the days of ASDF 1, a convenient hook was added to ASDF
so that when you call `cl:require`, it would try to load the named system,
and if not fall back on the builtin require mechanism.
This was a cool hack, and when you merely wanted a dependency, it was easier to type
`(require :foo)` than `(asdf:operate 'asdf:load-op :foo)` as you then had to.
Moreover, on SBCL where ASDF was developed,
ASDF itself was used to compile and load SBCL modules at runtime,
so this hook came naturally.

As ASDF maintainer, I now consider this in bad taste:

  * First, this hook is not 100% portable, so it is bad taste to recommend relying on it.
  * Second, `cl:require` has a mechanism for loading things only once by checking `cl:*modules*`
    which may subtly interfere with ASDF's mechanism for keeping things up to date.
  * Third, it interferes with ASDF's capacity to detect legitimate vs illegitimate
    recursive uses of `operate` at places that defeat tracking of dependencies.
  * Fourth, it adds a lot of complexity for dubious gain: at a time you had to type
    `(asdf:operate 'load-op :foo)`, `(require :foo)` may have been a nice short-hand,
    but it isn't such a great gain over `(asdf:make :foo)`.
  * Fifth, SBCL now uses ASDF 3's `compile-bundle-op` to create a fasl
    during the build of SBCL itself, and that fasl can latter be loaded at runtime without ASDF.
    Therefore the hook has no natural use anymore.

Similarly, in ASDF 2.21 I added a function `require-system`
that used to called `load-system` with `:force-not (already-loaded-systems)`,
which was a nice hack at the time, that I latter used as part of the `cl:require` hook.
That was all a big mistake, as `:force-not` interfere with the ability to keep a coherent plan
across recursive uses of `asdf:operate` as required by builds that involve `:defsystem-depends-on`
and other ASDF extensions.
These days, this function only checks whether the requested component is already loaded,
and if not calls `asdf:load-system` on it.
This function MUST only be used at the toplevel, never in a script or build.
It may be deprecated in a future version of ASDF.

### More Elaborate System Definitions

#### package-inferred-system

When you start writing large enough systems,
putting everything in one big package leads to a big mess:
it's hard to find what function is defined where, or should be defined where;
you invent your own symbol prefixing system to avoid name clashes;
totally unrelated things end up in the same mother-of-all package;
you divide your mother-of-all package into a few subpackages,
but as the software keeps growing each of these packages in turn becomes too big.

Meanwhile, as you grow large enough libraries, you find that you loading
a big library just to use a small part of it becomes a big hassle,
leading to code bloat, too much recompilation, too much re-testing,
and not enough understanding of what's going on.

A solution to both these problems is the "one file, one package, one system" style,
once spearheaded by faslpath and quick-build, and now available as part of ASDF
(since ASDF 3.1, 2014) using the `package-inferred-system` class.

Following this style, your top `.asd` file defines uses this class for its primary system
(you can still define secondary systems with different classes).
Then, any secondary system, if not explicitly defined,
will be searched for in a file as named by the secondary system suffix,
under the directory that contains the system definition file.

Thus, if `~/common-lisp/foobar-1.0/foobar.asd` defines
a primary system of class `package-inferred-system`,
but no secondary class `foobar/x/y`, then ASDF will look for a file
`~/common-lisp/foobar-1.0/x/y.lisp` to contain this system.
That file will be a regular Lisp file,
that will begin with a `defpackage` or `uiop:define-package` form.
(The latter form is more friendly to live upgrades,
but also allows to use a mix of packages with a priority on symbol conflicts,
or to reexport imported symbols.)
Dependencies for this system will be deduced from the `:use`,
`:import-form` and `:shadowing-import-from` clauses of that `defpackage`,
where each package name is downcased and interpreted as a system name,
unless registered otherwise via `asdf:register-system-packages`.

This allows for large modular libraries, wherein you may use one file,
and only that file and its transitive dependencies will be loaded,
rather than the entire humongous library.

This also helps you enforce a discipline wherein it is always clear in which file
each symbol is defined, which files have symbols used by any other file, etc.

#### Using ASDF Extensions

If you need an ASDF extension, the recommended way is to use `:defsystem-depends-on`.
The extension will define new classes of operations or components, new functions, etc.
If it defines them in the ASDF package, you can refer to them using a keyword of the same name,
e.g. `:cffi-wrapper-file` for one of the component classes defined by system `cffi-grovel`,
`:static-program-op` for one of the operation classes defined by `cffi-toolchain`, or
`:f2cl-system` for the system class defined by system `f2cl`.
Otherwise, you can refer to them using a string that when read yields the symbol, e.g.
the system `asdf-finalizers` defines a function that you can use as an around-compile hook
for your system using the clause `:around-compile "asdf-finalizers:check-finalizers-around-compile"`.
You cannot usually specify the qualified symbol directy, because it lives in a package
that hasn't been defined yet at the time the `defsystem` form is read.

Sometimes, you may want to use several build extensions at once,
but a given system, smaller component or operation, can have only one class.
You may split that system in two (say, a primary system and a secondary system)
so each has its own extension (say, make the primary system a `package-inferred-system`,
but have a secondary system be a `program-system`).
For more advanced cases, you may have to define a class that inherits from multiple other classes
in your own extension.
(In a prototype OO system, you could just mix and match extensions without defining a new class,
but CLOS is not a prototype OO system.)

#### Code in .asd files

ASDF currently allows arbitrary Lisp code in a `.asd` file.
I would like to deprecate that in the future to make ASDF more declarative.
In the meantime, here are some guidelines for how to write such code properly.

714 715 716 717 718
First, you SHOULD minimize the amount of such code and
you SHOULD strongly consider defining an extension
in a separate `.asd` file that you `:defsystem-depends-on`.

Then, `.asd` files are read with the current package being `asdf-user`.
719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798
You MAY use any symbols defined in packages `cl`, `asdf` and `uiop`,
and you SHOULD stay in package `asdf-user` if it suffices;
you SHOULD NOT needlessly create a package for you system definition file
if you're not going to define any such function nor variable nor macro.
But you MUST NOT pollute that package with bindings that could clash with uses by other systems.

It is poor taste to define functions, variables or macros in package `asdf-user`,
unless strictly necessary or widely useful, and even then with a long name
that distinctively includes the name of your system.
However it is perfectly acceptable to define methods on existing functions,
preferrably using the `:perform` syntax used above, as part of a `defsystem` form.

If you must define new functions, variables or macros,
you MUST define a new package in which to define them.
That system would typically be named `foobar-system`
and would export the relevant symbols, as in:

    (defpackage :foobar-system
      (:use :cl :asdf :uiop)
      (:export #:my-foobar-class ...))
    (in-package :foobar-system)

That said, if you need more than trivial definitions,
and if these definitions ever need to be used by others,
it is appropriate to move them to their own system.

Also, it is poor taste to define symbols used in the system itself (e.g. `*version*`)
since the system then depends on ASDF and cannot be directly built using Bazel,
or some other future build system.
For versions, consider having ASDF extract the version from a file, as in
`:version (:read-file-form "variables.lisp" :at (2 2 2)`.
See the ASDF manual for details, and `asdf.asd` itself for an example.

#### Conditional Code

The recommended way to conditionally use code is to rely on the CL features;
yet, you SHOULD NOT use read-time conditionals if you can avoid it,
and instead use the ASDF 3 `:feature` and `:if-feature` features:

  * To conditionally depend on another system or module,
    specify as a `:depends-on` argument something like `(:feature :sbcl (:require :sb-introspect))`
    or `(:feature (:or :ccl :sbcl) "bordeaux-threads")`.
  * To conditionally enable a component,
    specify a `:if-feature` argument to it like `(:file "sbcl-support" :if-feature :sbcl)`

Note that the feature expressions MUST use keywords, not symbols in the current package
(unless you're pervert enough to make `:keyword` your current package).

Also note that ASDF at this time wants all the component classes to be defined,
so if your component classes are conditionally defined, you may have to resort back
to the otherwise deprecated use of read-time conditionals, as in:

    (defsystem "foo"
      :defsystem-depends-on ((:feature :sbcl "some-sb-magic"))
      :components (#+sbcl (:sb-magic-file "some-magic")))

If the condition you want to express doesn't have a corresponding `*features*` keyword,
you MAY use a `(when ... (pushnew :my-keyword *features*))` form to add a feature keyword for it.
However, you SHOULD NOT needlessly push new features;
in particular, the mere fact of have loaded your system does not warrant a new feature.
A feature is only warranted if your system is some deep infrastructure
with mostly compatible rivals that it needs to be distinguished from.

### More Elaborate Testing

You MUST NOT call `asdf:operate` or any of its derivatives,
such as `asdf:load-system` or `asdf:test-system` from within a `perform` method.
Instead, you SHOULD declare proper dependencies between actions using methods on
`component-depends-on`, or more simply using an `:in-order-to` clause in your `defsystem` form.

You MUST NOT call `asdf:clear-system` or in any way interfere with the build
while in a `perform` method.

You SHOULD NOT define methods on `asdf:operate` --- most of the time it's totally the wrong thing
because users would be "operating" on your system, but on their system that depends on it.
Instead you SHOULD define methods on `asdf:perform`, `asdf:component-depends-on`, etc.

### Defining ASDF Extensions

#### Source File Types
799

800 801 802 803 804
You MUST NOT define methods on `source-file-type`. This bad ASDF 1 interface must die.
Instead, override the slot `type` for your file class and provide a proper `:initform`, as in:

    (defclass cl-source-file.l (cl-source-file) ((type :initform "l")))

805 806 807
You can then provide the name of that class as class name for individual components,
or as `:default-component-class` to relevant systems or modules.

808 809 810 811 812 813 814 815 816 817 818 819 820 821
#### Conditional Outputs and Conditional Perform

An operation MUST actually create all the outputs declared by the `output-files` method.
If some of these outputs are conditional, the `output-files` method MUST check the condition
and remove uncreated outputs from the list.
If some output is sometimes non meaningful but that is only known after calling the `perform` method,
it is sometimes appropriate to create an empty file instead of not creating the file.

If you conditionally avoid performing an action altogether using an `:around` method,
you MUST still call `mark-operation-done` on the action in the branch of the method
where the action is not performed.
In that case, both the `output-files` and `input-files` methods
should probably return `nil` when that condition is met.
Alternatively, if avoiding some actions is uniform for all
822
operations on a given component, you MAY want to override or wrap around
823 824 825
the `component-if-feature` method for that component instead of defining
all those `perform`, `output-files` and `input-files` methods.

826 827 828 829 830 831 832 833 834 835 836
## Other

Do not side-effect the current `*readtable*`,
which is never guaranteed to be writable, and
may not be what is current when you need the modification.
If you need modifications to the readtable, use `named-readtables`,
`reader-interception`, or some other adequate mechanism.
To bind the readtable, use `named-readtables:in-readtable` or `cl-syntax:in-syntax`
in each file that needs a non-standard readtable, and/or use an `:around-compile` hook
to automatically bind it around every file in a module or system.