Skip to content
Snippets Groups Projects
asdf.texinfo 132 KiB
Newer Older
The documentation was grossly out of date.


@subsection ASDF itself is versioned

Between new features, old bugs fixed, and new bugs introduced,
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
there were various releases of ASDF in the wild,
and no simple way to check which release had which feature set.
People using or writing systems had to either make worst-case assumptions
as to what features were available and worked,
or take great pains to have the correct version of ASDF installed.

With ASDF 2, we provide a new stable set of working features
that everyone can rely on from now on.
Use @code{#+asdf2} to detect presence of ASDF 2,
@code{(asdf:version-satisfies (asdf:asdf-version) "2.000")}
to check the availability of a version no earlier than required.

@subsection ASDF can be upgraded

When an old version of ASDF was loaded,
it was very hard to upgrade ASDF in your current image
without breaking everything.
Instead you have to exit the Lisp process and
somehow arrange to start a new one from a simpler image.
Something that can't be done from within Lisp,
making automation of it difficult,
which compounded with difficulty in configuration,
made the task quite hard.
Yet as we saw before, the task would have been required
to not have to live with the worst case or non-portable
subset of ASDF features.

With ASDF 2, it is easy to upgrade
from ASDF 2 to later versions from within Lisp,
and not too hard to upgrade from ASDF 1 to ASDF 2 from within Lisp.
We support hot upgrade of ASDF and any breakage is a bug
that we will do our best to fix.
There are still limitations on upgrade, though,
most notably the fact that after you upgrade ASDF,
you must also reload or upgrade all ASDF extensions.

@subsection Decoupled release cycle

When vendors were releasing their Lisp implementations with ASDF,
they had to basically never change version
because neither upgrade nor downgrade was possible
without breaking something for someone,
and no obvious upgrade path was visible and recommendable.

With ASDF 2, upgrade is possible, easy and can be recommended.
This means that vendors can safely ship a recent version of ASDF,
confident that if a user isn't fully satisfied,
he can easily upgrade ASDF and deal
with a supported recent version of it.
This means that release cycles will be causally decoupled,
the practical consequence of which will mean faster convergence
towards the latest version for everyone.

@subsection Pitfalls of the transition to ASDF 2

The main pitfalls in upgrading to ASDF 2 seem to be related
to the output translation mechanism.

@itemize

@item
Output translations is enabled by default. This may surprise some users,
most of them in pleasant way (we hope), a few of them in an unpleasant way.
It is trivial to disable output translations.
@xref{FAQ,,``How can I wholly disable the compiler output cache?''}.

@item
Some systems in the large have been known not to play well with output translations.
They were relatively easy to fix.
Once again, it is also easy to disable output translations,
or to override its configuration.

@item
The new ASDF output translations are incompatible with ASDF-Binary-Locations.
They replace A-B-L, and there is compatibility mode to emulate
your previous A-B-L configuration.
See @code{enable-asdf-binary-locations-compatibility} in
@pxref{Controlling where ASDF saves compiled files,,Backward Compatibility}.
But thou shall not load ABL on top of ASDF 2.

@item
ASDF pathname designators are now specified in places where they were unspecified,
and a few small adjustments have to be made to some non-portable defsystems.
Notably, in the @code{:pathname} argument to a @code{defsystem} and its components,
a logical pathname (or implementation-dependent hierarchical pathname)
must now be specified with @code{#p} syntax
where the namestring might have previously sufficed;
moreover when evaluation is desired @code{#.} must be used,
where it wasn't necessary in the toplevel @code{:pathname} argument.

@item
There is a slight performance bug, notably on SBCL,
when initially searching for @file{asd} files,
the implicit @code{(directory "/configured/path/**/*.asd")}
for every configured path @code{(:tree "/configured/path/")}
in your @code{source-registry} configuration can cause a slight pause.
Try to @code{(time (asdf:initialize-source-registry))}
to see how bad it is or isn't on your system.
If you insist on not having this pause,
you can avoid the pause by overriding the default source-registry configuration
and not use any deep @code{:tree} entry but only @code{:directory} entries
or shallow @code{:tree} entries.
Or you can fix your implementation to not be quite that slow
when recursing through directories.

@item
On Windows, only LispWorks supports proper default configuration pathnames
based on the Windows registry.
Other implementations make do with environment variables.
Windows support is somewhat less tested than Unix support.
Please help report and fix bugs.

@item
The mechanism by which one customizes a system so that Lisp files
may use a different extension from the default @file{.lisp} has changed.
Previously, the pathname for a component was lazily computed when operating on a system,
and you would
@code{(defmethod source-file-type ((component cl-source-file) (system (eql (find-system 'foo))))
  (declare (ignorable component system)) "cl")}.
Now, the pathname for a component is eagerly computed when defining the system,
Robert P. Goldman's avatar
Robert P. Goldman committed
and instead you will @code{(defclass my-cl-source-file (cl-source-file) ((type :initform "cl")))}
and use @code{:default-component-class my-cl-source-file} as argument to @code{defsystem},
as detailed in a @pxref{FAQ,How do I create a system definition where all the source files have a .cl extension?} below.

@findex source-file-type

@section Issues with installing the proper version of ASDF

@subsection ``My Common Lisp implementation comes with an outdated version of ASDF. What to do?''

We recommend you upgrade ASDF.
@xref{Loading ASDF,,Upgrading ASDF}.
If this does not work, it is a bug, and you should report it.
@xref{FAQ, report-bugs, Where do I report a bug}.
In the meantime, you can load @file{asdf.lisp} directly.
@xref{Loading ASDF,Loading an otherwise installed ASDF}.
@subsection ``I'm a Common Lisp implementation vendor. When and how should I upgrade ASDF?''
Starting with current candidate releases of ASDF 2,
it should always be a good time to upgrade to a recent ASDF.
You may consult with the maintainer for which specific version they recommend,
but the latest @code{release} should be correct.
We trust you to thoroughly test it with your implementation before you release it.
If there are any issues with the current release,
it's a bug that you should report upstream and that we will fix ASAP.

As to how to include ASDF, we recommend the following:
If ASDF isn't loaded yet, then @code{(require :asdf)}
should load the version of ASDF that is bundled with your system.
You may have it load some other version configured by the user,
if you allow such configuration.

@item
If your system provides a mechanism to hook into @code{CL:REQUIRE},
then it would be nice to add ASDF to this hook the same way that
ABCL, CCL, CLISP, CMUCL, ECL, SBCL and SCL do it.

@item
You may, like SBCL, have ASDF be implicitly used to require systems
that are bundled with your Lisp distribution.
If you do have a few magic systems that come with your implementation
in a precompiled way such that one should only use the binary version
that goes with your distribution, like SBCL does,
then you should add them in the beginning of @code{wrapping-source-registry}.

@item
If you have magic systems as above, like SBCL does,
then we explicitly ask you to @emph{NOT} distribute
@file{asdf.asd} as part of those magic systems.
You should still include the file @file{asdf.lisp} in your source distribution
and precompile it in your binary distribution,
but @file{asdf.asd} if included at all,
should be secluded from the magic systems,
in a separate file hierarchy;
alternatively, you may provide the system
after renaming it and its @file{.asd} file to e.g.
@code{asdf-ecl} and @file{asdf-ecl.asd}, or
@code{sb-asdf} and @file{sb-asdf.asd}.
Indeed, if you made @file{asdf.asd} a magic system,
then users would no longer be able to upgrade ASDF using ASDF itself
to some version of their preference that
they maintain independently from your Lisp distribution.

@item
If you do not have any such magic systems, or have other non-magic systems
that you want to bundle with your implementation,
then you may add them to the @code{default-source-registry},
and you are welcome to include @file{asdf.asd} amongst them.

@item
Please send us upstream any patches you make to ASDF itself,
so we can merge them back in for the benefit of your users
when they upgrade to the upstream version.


@section Issues with configuring ASDF

@subsection ``How can I customize where fasl files are stored?''

@xref{Controlling where ASDF saves compiled files}.

Note that in the past there was an add-on to ASDF called
@code{ASDF-binary-locations}, developed by Gary King.
That add-on has been merged into ASDF proper,
then superseded by the @code{asdf-output-translations} facility.

Note that use of @code{asdf-output-translations}
can interfere with one aspect of your systems
--- if your system uses @code{*load-truename*} to find files
(e.g., if you have some data files stored with your program),
then the relocation that this ASDF customization performs
is likely to interfere.
Use @code{asdf:system-relative-pathname} to locate a file
in the source directory of some system, and
use @code{asdf:apply-output-translations} to locate a file
whose pathname has been translated by the facility.
@subsection ``How can I wholly disable the compiler output cache?''

To permanently disable the compiler output cache
for all future runs of ASDF, you can:

@example
mkdir -p ~/.config/common-lisp/asdf-output-translations.conf.d/
echo ':disable-cache' > ~/.config/common-lisp/asdf-output-translations.conf.d/99-disable-cache.conf
@end example

This assumes that you didn't otherwise configure the ASDF files
(if you did, edit them again),
and don't somehow override the configuration at runtime
with a shell variable (see below) or some other runtime command
(e.g. some call to @code{asdf:initialize-output-translations}).

To disable the compiler output cache in Lisp processes
run by your current shell, try (assuming @code{bash} or @code{zsh})
(on Unix and cygwin only):

@example
export ASDF_OUTPUT_TRANSLATIONS=/:
@end example

To disable the compiler output cache just in the current Lisp process,
use (after loading ASDF but before using it):

@example
@section Issues with using and extending ASDF to define systems

@subsection ``How can I cater for unit-testing in my system?''

ASDF provides a predefined test operation, @code{test-op}.
@xref{Predefined operations of ASDF, test-op}.
The test operation, however, is largely left to the system definer to specify.
@code{test-op} has been
a topic of considerable discussion on the
@url{http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel,asdf-devel mailing list},
and on the
@url{https://launchpad.net/asdf,launchpad bug-tracker}.
Here are some guidelines:

@itemize
@item
For a given system, @var{foo}, you will want to define a corresponding
test system, such as @var{foo-test}.  The reason that you will want this
separate system is that ASDF does not out of the box supply components
that are conditionally loaded.  So if you want to have source files
(with the test definitions) that will not be loaded except when testing,
they should be put elsewhere.

@item
The @var{foo-test} system can be defined in an asd file of its own or
together with @var{foo}.  An aesthetic preference against cluttering up
the filesystem with extra asd files should be balanced against the
question of whether one might want to directly load @var{foo-test}.
Typically one would not want to do this except in early stages of
debugging.

@item
Record that testing is implemented by @var{foo-test}.  For example:
@example
(defsystem @var{foo}
   :in-order-to ((test-op (test-op @var{foo-test})))
   ....)

(defsystem @var{foo-test}
   :depends-on (@var{foo} @var{my-test-library} ...)
   ....)
@end example
@end itemize

This procedure will allow you to support users who do not wish to
install your test framework.

One oddity of ASDF is that @code{operate} (@pxref{Operations,operate})
does not return a value.  So in current versions of ASDF there is no
reliable programmatic means of determining whether or not a set of tests
has passed, or which tests have failed.  The user must simply read the
console output.  This limitation has been the subject of much
discussion.

@subsection ``How can I cater for documentation generation in my system?''
The ASDF developers are currently working to add a @code{doc-op}
to the set of predefined ASDF operations.
@xref{Predefined operations of ASDF}.
See also @url{https://bugs.launchpad.net/asdf/+bug/479470}.
@subsection ``How can I maintain non-Lisp (e.g. C) source files?''
See @code{cffi}'s @code{cffi-grovel}.
@anchor{report-bugs}
@subsection ``I want to put my module's files at the top level.  How do I do this?''
By default, the files contained in an asdf module go
in a subdirectory with the same name as the module.
However, this can be overridden by adding a @code{:pathname ""} argument
For example, here is how it could be done
in the spatial-trees ASDF system definition for ASDF 2:
Robert P. Goldman's avatar
Robert P. Goldman committed

@example
(asdf:defsystem :spatial-trees
  :components
  ((:module base
Robert P. Goldman's avatar
Robert P. Goldman committed
            :components
            ((:file "package")
             (:file "basedefs" :depends-on ("package"))
             (:file "rectangles" :depends-on ("package"))))
   (:module tree-impls
            :depends-on (base)
Robert P. Goldman's avatar
Robert P. Goldman committed
            :components
            ((:file "r-trees")
             (:file "greene-trees" :depends-on ("r-trees"))
             (:file "rstar-trees" :depends-on ("r-trees"))
             (:file "rplus-trees" :depends-on ("r-trees"))
             (:file "x-trees" :depends-on ("r-trees" "rstar-trees"))))
   (:module viz
            :depends-on (base)
Robert P. Goldman's avatar
Robert P. Goldman committed
            :components
            ((:static-file "spatial-tree-viz.lisp")))
Robert P. Goldman's avatar
Robert P. Goldman committed
   (:module tests
            :depends-on (base)
Robert P. Goldman's avatar
Robert P. Goldman committed
            :components
            ((:static-file "spatial-tree-test.lisp")))
Robert P. Goldman's avatar
Robert P. Goldman committed
   (:static-file "LICENCE")
   (:static-file "TODO")))
@end example

All of the files in the @code{tree-impls} module are at the top level,
instead of in a @file{tree-impls/} subdirectory.
Note that the argument to @code{:pathname} can be either a pathname object or a string.
A pathname object can be constructed with the @file{#p"foo/bar/"} syntax,
but this is discouraged because the results of parsing a namestring are not portable.
A pathname can only be portably constructed with such syntax as
@code{#.(make-pathname :directory '(:relative "foo" "bar"))},
and similarly the current directory can only be portably specified as
@code{#.(make-pathname :directory '(:relative))}.
However, as of ASDF 2, you can portably use a string to denote a pathname.
The string will be parsed as a @code{/}-separated path from the current directory,
such that the empty string @code{""} denotes the current directory, and
@code{"foo/bar"} (no trailing @code{/} required in the case of modules)
portably denotes the same subdirectory as above.
When files are specified, the last @code{/}-separated component is interpreted
either as the name component of a pathname
(if the component class specifies a pathname type),
or as a name component plus optional dot-separated type component
(if the component class doesn't specifies a pathname type).
@subsection How do I create a system definition where all the source files have a .cl extension?

First, create a new @code{cl-source-file} subclass that provides an
initform for the @code{type} slot:

@lisp
(defclass my-cl-source-file (cl-source-file)
   ((type :initform "cl")))
@end lisp

To support both ASDF 1 and ASDF 2,
you may omit the above @code{type} slot definition and instead define:

@lisp
(defmethod source-file-type ((f my-cl-source-file) (m module))
  (declare (ignorable f m))
  "cl")
@end lisp

Then make your system use this subclass in preference to the standard
one:

@lisp
(defsystem my-cl-system
  :default-component-class my-cl-source-file
   ....
)
@end lisp

We assume that these definitions are loaded into a package that uses
@code{ASDF}.


@node  TODO list, Inspiration, FAQ, Top
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
@comment  node-name,  next,  previous,  up
Here is an old list of things to do,
in addition to the bugs that are now tracked on launchpad:
@url{https://launchpad.net/asdf}.

@section Outstanding spec questions, things to add

** packaging systems

*** manual page component?

** style guide for .asd files

You should either use keywords or be careful
with the package that you evaluate defsystem forms in.
Otherwise @code{(defsystem partition ...)}
being read in the @code{cl-user} package
will intern a @code{cl-user:partition} symbol,
which will then collide with the @code{partition:partition} symbol.

Actually there's a hairier packages problem to think about too.
@code{in-order-to} is not a keyword:
if you read @code{defsystem} forms in a package that doesn't use ASDF,
odd things might happen.


** extending defsystem with new options

You might not want to write a whole parser,
but just to add options to the existing syntax.
Reinstate @code{parse-option} or something akin.


** document all the error classes

** what to do with compile-file failure

Should check the primary return value from compile-file and see if
that gets us any closer to a sensible error handling strategy

** foreign files

lift unix-dso stuff from db-sockets

** Diagnostics

A ``dry run'' of an operation can be made with the following form:

@lisp
(traverse (make-instance '<operation-name>)
          (find-system <system-name>)
          'explain)
@end lisp

This uses unexported symbols.
What would be a nice interface for this functionality?
@section Missing bits in implementation

** reuse the same scratch package whenever a system is reloaded from disk

** proclamations probably aren't

** when a system is reloaded with fewer components than it previously had, odd things happen
We should do something inventive when processing a @code{defsystem} form,
like take the list of kids and @code{setf} the slot to @code{nil},
then transfer children from old to new list as they're found.

** (stuff that might happen later)

*** Propagation of the @code{:force} option.
``I notice that
        @code{(asdf:compile-system :araneida :force t)}
also forces compilation of every other system the @code{:araneida} system depends on.
This is rarely useful to me;
usually, when I want to force recompilation of something more than a single source file,
I want to recompile only one system.
So it would be more useful to have @code{make-sub-operation}
refuse to propagate @code{:force t} to other systems, and
propagate only something like @code{:force :recursively}.

Ideally what we actually want is some kind of criterion that says
to which systems (and which operations) a @code{:force} switch will propagate.
The problem is perhaps that ``force'' is a pretty meaningless concept.
How obvious is it that @code{load :force t} should force @emph{compilation}?
But we don't really have the right dependency setup
for the user to compile @code{:force t} and expect it to work
(files will not be loaded after compilation, so the compile
environment for subsequent files will be emptier than it needs to be)

What does the user actually want to do when he forces?
Usually, for me, update for use with a new version of the Lisp compiler.
Perhaps for recovery when he suspects that something has gone wrong.
Or else when he's changed compilation options or configuration
in some way that's not reflected in the dependency graph.
Other possible interface: have a ``revert'' function akin to @code{make clean}.
(asdf:revert 'asdf:compile-op 'araneida)
would delete any files produced by @code{(compile-system :araneida)}.
Of course, it wouldn't be able to do much about stuff in the image itself.
@code{traverse}
There's a difference between a module's dependencies (peers)
and its components (children).
Perhaps there's a similar difference in operations?
For example, @code{(load "use") depends-on (load "macros")} is a peer,
whereas @code{(load "use") depends-on (compile "use")}
is more of a ``subservient'' relationship.
@node  Inspiration, Concept Index, TODO list, Top
@comment  node-name,  next,  previous,  up
@chapter Inspiration

@section mk-defsystem (defsystem-3.x)

We aim to solve basically the same problems as @code{mk-defsystem} does.
However, our architecture for extensibility
better exploits CL language features (and is documented),
and we intend to be portable rather than just widely-ported.
No slight on the @code{mk-defsystem} authors and maintainers is intended here;
that implementation has the unenviable task
of supporting pre-ANSI implementations, which is no longer necessary.

The surface defsystem syntax of asdf is more-or-less compatible with
@code{mk-defsystem}, except that we do not support
the @code{source-foo} and @code{binary-foo} prefixes
for separating source and binary files, and
we advise the removal of all options to specify pathnames.
The @code{mk-defsystem} code for topologically sorting
a module's dependency list was very useful.

@section defsystem-4 proposal

Marco and Peter's proposal for defsystem 4 served as the driver for
many of the features in here.  Notable differences are:

@itemize
@item
We don't specify output files or output file extensions
as part of the system.
If you want to find out what files an operation would create,
ask the operation.

@item
We don't deal with CL packages

If you want to compile in a particular package, use an @code{in-package} form
in that file (ilisp / SLIME will like you more if you do this anyway)

@item
There is no proposal here that @code{defsystem} does version control.

A system has a given version which can be used to check dependencies,
but that's all.
@end itemize

The defsystem 4 proposal tends to look more at the external features,
whereas this one centres on a protocol for system introspection.

@section kmp's ``The Description of Large Systems'', MIT AI Memo 801
Available in updated-for-CL form on the web at
Robert P. Goldman's avatar
Robert P. Goldman committed
@url{http://nhplace.com/kent/Papers/Large-Systems.html}
In our implementation we borrow kmp's overall @code{PROCESS-OPTIONS}
and concept to deal with creating component trees
from @code{defsystem} surface syntax.
[ this is not true right now, though it used to be and
probably will be again soon ]


@c -------------------


@node Concept Index, Function and Class Index, Inspiration, Top
@unnumbered Concept Index
@printindex cp

@node Function and Class Index, Variable Index, Concept Index, Top
@unnumbered Function and Class Index
@printindex fn

@node Variable Index,  , Function and Class Index, Top
@unnumbered Variable Index