Commit 7dbc8dd8 authored by Francois-Rene Rideau's avatar Francois-Rene Rideau
Browse files
parents 1f13e432 2aa7d887
......@@ -20,7 +20,7 @@ according to a model implemented using the {\longCLOS} ({\CLOS}).
This model is exposed to programmers and extensible by programmers,
who may thereby adapt {\ASDF} to their needs.
For most {\ASDF} users, the one important service that {\ASDF} provides is
For most {\ASDF} users, the service that {\ASDF} provides is
the \lisp{asdf:load-system} function.
When {\ASDF} is properly configured,
\lisp{(asdf:load-system "foo")} will load the specified system into the current image;
......@@ -40,8 +40,8 @@ To that end, it will take the following steps:
(3) execute this plan.
%\end{itemize}
Additionally, system maintainers find themselves writing their own system definitions
in order to manage their own software.
In addition to loading others' systems, system builders will write
their own system definitions.
More advanced developers extend {\ASDF}'s object model:
they may define new {\em components} in addition to Lisp source files,
such as C source files or protocol buffer definitions;
......@@ -50,10 +50,19 @@ such as documentation generation or regression testing.
% Unfortunately, {\ASDF} has oddities that have, all too often,
% hindered their ability to use {\ASDF} as a tool.
{\ASDF} provides software management for {\CL}.
It is most notoriously used to build {\CL} software;
but it can also help with a wider range of software management tasks,
such as testing, documentation or packaging.
% I cut all of the following --- it's redundant because in the next section we
% explain that ASDF does more than make does, in just the ways we explain here.
% {\ASDF} provides software management for {\CL}.
% \rtof{I'm sorry to be a nitpicker, but I know we've gone around on this before.
% ```Notoriously'' is just the wrong word here. Here's the wordnet definition:
% ``ill-famed, infamous, notorious (known widely and usually unfavorably) "a
% notorious gangster"; "the tenderloin district was notorious for vice"; "the
% infamous Benedict Arnold";''}
% Not only can it be used to build {\CL} software;
% it can also help with a wider range of software management tasks,
% such as testing, documentation or packaging.
{\ASDF} is a central piece of software for the {\CL} community:
{\CL} libraries --- and especially the open source libraries ---
are overwhelmingly delivered with {\ASDF} system descriptions, and
......@@ -75,11 +84,12 @@ they otherwise differ in their goals, their design,
the constraints they respect, their internal architecture,
the concepts that underly them, the interfaces they expose to users.
Hence, {\ASDF} finds systems and loads systems,
\fixme{We use ``analogous'' too many times in the following paragraph.}
{\ASDF} finds systems and loads systems,
problems that are not handled
by build tools such as {\make} or {\ant}.
As for finding systems,
these tools must be pointed at a system definition
These other build tools don't search for systems; they
must be pointed at a system definition
in the current or specified directory.
Finding systems at build time would be analogous
to a subset of \texttt{libtool};
......@@ -98,15 +108,18 @@ If system components have been changed, it is {\ASDF}'s job,
at the user's request, to generate and execute plans
to modify previously-loaded systems in order to accommodate those changes.
{\ASDF} also differs significantly from {\make}
{\ASDF} also differs from {\make}
in terms of how systems are specified.
{\make} is built around a complex combination of
multiple layers of specialized and generalized languages:
a text substitution program is expanded into
simple pattern matching rules used by an inference engine
that will run simply parameterized shell scripts.
multiple layers of languages --- some domain-specific languages and some
generalized programming languages --- used in \emph{makefiles}.
% \rtof{I would like to cut the following; I think it's enough that the reader
% knows that there are multiple languages involved.}
% a text substitution program is expanded into
% simple pattern matching rules used by an inference engine
% that will run simply parameterized shell scripts.
{\make} is a powerful tool that can express arbitrary programs,
but the result is a mesh of code that defies any simple analysis.
but makefiles can be a mesh of code that defies any simple analysis.
{\ASDF} is a small {\CL} program,
and its system definitions are data rather than programs.
Any non-trivial code required by an {\ASDF} system definition
......@@ -211,81 +224,131 @@ The system definition search is one of the aspects of {\ASDFi} that we reformed;
the previous protocol had to be initialized using arbitrary Lisp code,
and had a number of undesirable features.
See Section \ref{sec:input-locations}.
{\operate} then generates a \emph{plan} for completing the {\operation}
After a system definition is found,
{\operate} generates a \emph{plan} for completing the {\operation}
using the function {\traverse} described below.
The plan will be a list of steps,
each step a pair of an \emph{operation} and a \emph{component};
The plan will be a list of {\step}s,
each {\step} a pair of an \emph{operation} and a \emph{component};
the list will be topologically sorted such that
all the dependencies of a step appear before that step.
all the dependencies of a {\step} appear before that {\step}.
Finally, if a plan is successfully generated,
{\operate} will {\perform} each of its steps in sequence.
If a step fails, {\ASDF} offers the user a chance to
correct the step, retry, and continue according to plan,
making it easy to fix simple mistakes without interrupting the build.
{\operate} will {\perform} each of its {\step}s in sequence.
If a {\step} fails, {\ASDF} offers the user a chance to
correct the {\step}, retry, and continue according to plan,
making it easy to fix simple mistakes without interrupting the operation.
\subsection{Plan generation}
\label{sec:plan-generation}
The semantics of an {\ASDF} system definition only specifies
ordering dependencies between build steps
of performing a given operation on a given component.
It doesn't specify a unique possible plan, or
even that the system compiler and loader should generate
a complete plan ahead of performing any step.
Indeed, a parallelizing extension to {\ASDF}, {\POIU},
does things differently (see below).
\rtof{I moved the material that was in your first paragraph here down to the end
of the section.
That material was all \emph{correct} and good to have, but putting it in front
amounted to presenting all of the exceptions and hedges first, before the
big picture explanation.}
However, it sometimes helps to understand how {\ASDF}
generates then executes plans, because it is a place where
the abstraction layer is all too often leaky.
{\traverse} performs a depth-first, postorder traversal
over the steps needed to operate on the target system and its dependencies.
Each step is a pair of an {\operation} and a {\component}.
over the {\step}s needed to operate on the target system and its dependencies.
Each {\step} is a pair of an {\operation} and a {\component}.
What we mean by postorder here is that
when applying an {\operation} to a {\system} (or {\module}),
the plan will contain steps
the plan will contain {\step}s
first to complete the {\operation} to the sub-components,
then to perform it on the {\system} (or {\module}) itself.
{\traverse} distinguishes between
\lisp{in-order-to} dependencies that should trigger a rebuild
and \lisp{do-first} dependencies that don't.
For instance, consider some component \texttt{a}
that \lisp{:depends-on} a component \texttt{b},
both being the default component type a \lisp{cl-source-file}.
Before \texttt{a} is either compiled (via operation \lisp{compile-op})
or loaded (via operation \lisp{load-op}),
\texttt{b} must first be loaded,
which itself depends on \texttt{b} having been compiled;
however, merely having to load \texttt{b} will not by itself
force the (re)compilation of \texttt{a},
but having to (re)compile \texttt{b} will
force the (re)compilation of \texttt{a}.
Therefore, {\traverse} first issues \lisp{in-order-to} dependencies,
and {\opdonep} status, and
if the operation needs to be done on the component,
issues \lisp{do-first} dependencies;
finally it issues the current operation and component pair
as steps of the plan.
While building a plan, {\traverse} avoids issuing unnecessary steps
and forcing steps that have \lisp{in-order-to} dependencies on them.
To determine whether a step not otherwise forced by a dependency
then to perform it on the {\system} (or {\module}) itself. For example, a load
op plan for the system in Figure~\ref{fig:sampleASD} is given as
Figure~\ref{fig:samplePlan}.
Note the lines marked with a \dag, which show that operations on composite components
(\module{}s and \system{}s) are scheduled after operations on their components.
\begin{figure}[t]
\begin{minipage}{1.0\columnwidth}
\begin{alltt}
((#<COMPILE-OP> . #<CL-SOURCE-FILE "packages">)
(#<LOAD-OP> . #<CL-SOURCE-FILE "packages">)
(#<COMPILE-OP> . #<CL-SOURCE-FILE "macros">)
(#<COMPILE-OP> . #<CL-SOURCE-FILE "classes">)
(#<LOAD-OP> . #<CL-SOURCE-FILE "macros">)
(#<LOAD-OP> . #<CL-SOURCE-FILE "classes">)
(#<COMPILE-OP> . #<CL-SOURCE-FILE "methods">)
(#<COMPILE-OP> . #<CL-SOURCE-FILE "hello">)
(#<LOAD-OP> . #<CL-SOURCE-FILE "hello">)
(#<COMPILE-OP> . #<CL-SOURCE-FILE "goodbye">)
\dag (#<COMPILE-OP> . #<MODULE "main">)
(#<COMPILE-OP> . #<SYSTEM "hello-lisp">)
(#<LOAD-OP> . #<CL-SOURCE-FILE "methods">)
(#<LOAD-OP> . #<CL-SOURCE-FILE "goodbye">)
\dag (#<LOAD-OP> . #<MODULE "main">)
\dag (#<LOAD-OP> . #<SYSTEM "hello-lisp">))
\end{alltt}
\end{minipage}
\centering
\caption{Sample \lisp{load-op} plan for the system definition in
Figure~\ref{fig:sampleASD}. We assume all files are previously compiled,
and \lisp{foo-utils} is already loaded.}
\label{fig:samplePlan}
\end{figure}
\rtof{I killed the explanation of the distinction between in-order-to and
do-first. I first tried to rewrite this to explain the two --- because we
just jumped into talking about these things which don't appear in our example --- but then that
seemed to require me to explain how \defsystem{} expands \lisp{:depends-on}
into in-order-to and do-first, and pretty soon I felt I was digging myself further
and further into the weeds. If you think it's absolutely necessary to explain
these two, then we can have another try, but I'd rather not.}
% The \lisp{:depends-on} specifications in \defsystem{} are expanded into two
% kinds of dependencies:
% \lisp{in-order-to} dependencies, where changes ``upstream'' (in the depended-on components)
% will trigger a rebuild in the ``downstream'' components,
% and \lisp{do-first} dependencies that don't.
% For instance, consider some component \texttt{a}
% that \lisp{:depends-on} a component \texttt{b},
% both being the default component type a \lisp{cl-source-file}.
% Before \texttt{a} is either compiled (via operation \lisp{compile-op})
% or loaded (via operation \lisp{load-op}),
% \texttt{b} must first be loaded,
% which itself depends on \texttt{b} having been compiled;
% however, merely having to load \texttt{b} will not by itself
% force the (re)compilation of \texttt{a},
% but having to (re)compile \texttt{b} will
% force the (re)compilation of \texttt{a}.
% Therefore, {\traverse} first issues \lisp{in-order-to} dependencies,
% and {\opdonep} status, and
% if the operation needs to be done on the component,
% issues \lisp{do-first} dependencies;
% finally it issues the current operation and component pair
% as {\step}s of the plan.
\rtof{I think the following is a bit confusing. Do we intend to say that we
avoid forcing {\step}s that have \lisp{in-order-to} dependencies? Or is it
trying to say ``we force {\step}s that have \lisp{in-order-to} dependencies while
avoiding unnecessary {\step}s? Here's my attempt to rewrite...}
While building a plan, {\traverse} attempts to avoid including steps that are
unnecessary.
Steps may be \emph{forced} by changes in depended-on components, but if they are
not forced, they will be left out if \traverse{} can determine that they have
already been done.
To determine whether a {\step}
has already been done,
it calls a generic function {\opdonep}.
For a compilation step, it compares the timestamps of
\traverse{} calls the generic function {\opdonep}.
For a compilation {\step}, \opdonep{} compares the timestamps of
the \lisp{input-files} and \lisp{output-files}
of the {\operation} and {\component}.
For a load step with \lisp{input-files} but no \lisp{output-files},
For a load {\step} with \lisp{input-files} but no \lisp{output-files},
it compares the timestamp of the input files
to the time it was last loaded into the current image (if ever).
Steps that are already up-to-date are skipped.
Programmers extending the \ASDF{} protocol to new operations or components may
define their own methods on \opdonep{}.
%{\Step}s that are already up-to-date are skipped.
Operations without either \lisp{input-files} nor \lisp{output-files}
(e.g. where the component is a {\module} or {\system})
(e.g., where the component is a {\module} or {\system})
are never considered necessary unless forced by their dependencies.
We will then see subtleties of this part of the protocol,
We will see subtleties of this part of the protocol,
when we discuss our modifications to {\traverse},
and the limitations on those modifications
(see Section \ref{sec:traverse-bug-fix}).
......@@ -295,7 +358,7 @@ and the limitations on those modifications
% before calling this done.}
Tracing {\traverse} and inspecting the plan it returns
is sometimes a good way to debug one's system definitions.
is often a good way to debug one's system definitions.
Running {\traverse} can also be useful as an introspection tool on a system
(e.g. to determine what files need be recompiled,
and run according tests after recompilation).
......@@ -309,6 +372,18 @@ but that has proven not to work,
because of the leakiness of the abstraction.
We discuss this further in Section \ref{sec:future-directions}.
Before we conclude the discussion, we would like to make two important
observations about the semantics of {\ASDF} system definitions.
These system definitions only specify
ordering dependencies between {\step}s.
The first observation is that, in general, these ordering dependencies only specify a partial order,
so for a given operation and system there may be multiple possible plans.
The second is that the semantics do not dictate a ``plan-then-execute'' implementation.
Indeed, a parallelizing extension to {\ASDF}, {\POIU},
does things differently (see Section \ref{poiu}).
\hide{
\ftor{
If some points of semantics are too complex, vague or ambiguous
......@@ -368,7 +443,7 @@ timestamps or the existence of files,
and does all these filesystem queries dynamically as the plan unfolds.
{\ASDF} queries the filesystem for timestamps once, in advance, while planning.
This difference has no visible consequence to the user in the common simple case;
but if a source file is generated by some of the steps it depends on,
but if a source file is generated by some of the {\step}s it depends on,
this will confuse {\ASDF} as it fails to find a timestamp at planning time.
The main benefit for {\make} is that it will work correctly when building things
in parallel (options \texttt{-j} or \texttt{-l} of {\make}).
......
......@@ -63,6 +63,9 @@
\newcommand{\CLOS}{CLOS}
\newcommand{\longCLOS}{Common Lisp Object System}
\newcommand{\step}{step}
\newcommand{\Step}{Step}
\newcommand{\ftor}[1]{\draft{{\textbf{{\fare} to {\rpg}}: #1}}}
\newcommand{\rtof}[1]{\draft{{\textbf{{\rpg} to {\fare}}: #1}}}
\newcommand{\fixme}[1]{\draft{{\textbf{FIXME}: #1}}}
......@@ -1033,7 +1036,7 @@ Release 2.006 is 144996 byte long;
the last version by Daniel Barlow, in 2004, was 38881 byte long.
}
\label{poiu}
On a positive note, {\CL} helped us keep things simple.
Andreas Fuchs wrote {\POIU}~\cite{QITAB},
an extension for {\ASDF} implementing parallel compilation.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment