Commit 63882ed4 authored by Francois-Rene Rideau's avatar Francois-Rene Rideau
Browse files

Another pass at asdf plan generation

parent 7dbc8dd8
......@@ -40,28 +40,13 @@ To that end, it will take the following steps:
(3) execute this plan.
%\end{itemize}
In addition to loading others' systems, system builders will write
their own system definitions.
Most developers load and modify existing systems.
Advanced developers define their own systems.
More advanced developers extend {\ASDF}'s object model:
they may define new {\em components} in addition to Lisp source files,
such as C source files or protocol buffer definitions;
or they may define new {\em operations} in addition to compiling and loading,
such as documentation generation or regression testing.
% Unfortunately, {\ASDF} has oddities that have, all too often,
% hindered their ability to use {\ASDF} as a tool.
% I cut all of the following --- it's redundant because in the next section we
% explain that ASDF does more than make does, in just the ways we explain here.
% {\ASDF} provides software management for {\CL}.
% \rtof{I'm sorry to be a nitpicker, but I know we've gone around on this before.
% ```Notoriously'' is just the wrong word here. Here's the wordnet definition:
% ``ill-famed, infamous, notorious (known widely and usually unfavorably) "a
% notorious gangster"; "the tenderloin district was notorious for vice"; "the
% infamous Benedict Arnold";''}
% Not only can it be used to build {\CL} software;
% it can also help with a wider range of software management tasks,
% such as testing, documentation or packaging.
{\ASDF} is a central piece of software for the {\CL} community:
{\CL} libraries --- and especially the open source libraries ---
......@@ -84,21 +69,20 @@ they otherwise differ in their goals, their design,
the constraints they respect, their internal architecture,
the concepts that underly them, the interfaces they expose to users.
\fixme{We use ``analogous'' too many times in the following paragraph.}
{\ASDF} finds systems and loads systems,
problems that are not handled
by build tools such as {\make} or {\ant}.
These other build tools don't search for systems; they
must be pointed at a system definition
These other build tools don't search for systems;
they must be pointed at a system definition
in the current or specified directory.
Finding systems at build time would be analogous
to a subset of \texttt{libtool};
finding and loading systems at runtime would be more analogous
to a subset of the Unix dynamic linker \texttt{ld.so}.
finding and loading systems at runtime would correspond to
a subset of the Unix dynamic linker \texttt{ld.so}.
As for loading,
the fact that {\ASDF} is available for interactive use
makes it analogous to some component of the operating system shell;
loading some systems might thus be analogous to
loading some systems might thus be similar to
importing shell functions, starting a daemon,
registering a plugin in your browser,
loading code into some master process, etc.
......@@ -111,13 +95,15 @@ to modify previously-loaded systems in order to accommodate those changes.
{\ASDF} also differs from {\make}
in terms of how systems are specified.
{\make} is built around a complex combination of
multiple layers of languages --- some domain-specific languages and some
generalized programming languages --- used in \emph{makefiles}.
multiple layers of languages --- some domain-specific languages and
some generalized programming languages.
{\make} interprets a \emph{makefile} thusly:
% \rtof{I would like to cut the following; I think it's enough that the reader
% knows that there are multiple languages involved.}
% a text substitution program is expanded into
% simple pattern matching rules used by an inference engine
% that will run simply parameterized shell scripts.
% \ftor{I think it brings some insight as to the nature of the difference}
a text substitution program is expanded into
simple pattern matching rules used by an inference engine
that will run simply parameterized shell scripts.
{\make} is a powerful tool that can express arbitrary programs,
but makefiles can be a mesh of code that defies any simple analysis.
{\ASDF} is a small {\CL} program,
......@@ -214,16 +200,20 @@ Systems can be recursively organized in a tree of {\module}s
that may or may not map to a similar tree of directories.
System management tasks consist in applying the generic function
{\operate} on parameters specifying an {\operation} and a {\system},
the latter of which may be designated by a string or a symbol.
{\operate} will first find and load the proper system definition, if necessary.
{\operate} on parameters specifying an {\operation} and a {\system}.
The {\operation} is either a {\loadOp},
specifying that a system or component is to be loaded into the current image,
or a {\compileOp}
specifying that a system or component is to be compiled into the filesystem.
The {\system} is designated by a string or a symbol, and
{\operate} will first find and load the thus named system definition.
This is done by the generic function \lisp{find-system}, which searches
for the system definition in a configurable system registry
(and its in-memory cache).
The system definition search is one of the aspects of {\ASDFi} that we reformed;
the previous protocol had to be initialized using arbitrary Lisp code,
and had a number of undesirable features.
See Section \ref{sec:input-locations}.
%the previous protocol had to be initialized using arbitrary Lisp code,
%and had a number of undesirable features.
see Section \ref{sec:input-locations}.
After a system definition is found,
{\operate} generates a \emph{plan} for completing the {\operation}
......@@ -242,32 +232,27 @@ making it easy to fix simple mistakes without interrupting the operation.
\subsection{Plan generation}
\label{sec:plan-generation}
\rtof{I moved the material that was in your first paragraph here down to the end
of the section.
That material was all \emph{correct} and good to have, but putting it in front
amounted to presenting all of the exceptions and hedges first, before the
big picture explanation.}
{\traverse} performs a depth-first, postorder traversal
The plan-generating function {\traverse}
performs a depth-first, postorder traversal
over the {\step}s needed to operate on the target system and its dependencies.
Each {\step} is a pair of an {\operation} and a {\component}.
Each {\step} is a pair of an {\operation} and a {\component},
What we mean by postorder here is that
when applying an {\operation} to a {\system} (or {\module}),
the plan will contain {\step}s
first to complete the {\operation} to the sub-components,
then to perform it on the {\system} (or {\module}) itself. For example, a load
op plan for the system in Figure~\ref{fig:sampleASD} is given as
Figure~\ref{fig:samplePlan}.
Note the lines marked with a \dag, which show that operations on composite components
(\module{}s and \system{}s) are scheduled after operations on their components.
then to perform it on the {\system} (or {\module}) itself.
For example, a {\loadOp} plan
for the system in Figure~\ref{fig:sampleASD}
is given as Figure~\ref{fig:samplePlan}.
Note the lines marked with a \dag,
which show that operations on composite components ({\module}s and {\system}s)
are scheduled after operations on their components.
\begin{figure}[t]
\begin{minipage}{1.0\columnwidth}
\begin{alltt}
((#<COMPILE-OP> . #<CL-SOURCE-FILE "packages">)
((#<COMPILE-OP> . #<CL-SOURCE-FILE "packages">)
(#<LOAD-OP> . #<CL-SOURCE-FILE "packages">)
(#<COMPILE-OP> . #<CL-SOURCE-FILE "macros">)
(#<COMPILE-OP> . #<CL-SOURCE-FILE "classes">)
......@@ -277,7 +262,7 @@ Note the lines marked with a \dag, which show that operations on composite compo
(#<COMPILE-OP> . #<CL-SOURCE-FILE "hello">)
(#<LOAD-OP> . #<CL-SOURCE-FILE "hello">)
(#<COMPILE-OP> . #<CL-SOURCE-FILE "goodbye">)
\dag (#<COMPILE-OP> . #<MODULE "main">)
\dag (#<COMPILE-OP> . #<MODULE "main">)
(#<COMPILE-OP> . #<SYSTEM "hello-lisp">)
(#<LOAD-OP> . #<CL-SOURCE-FILE "methods">)
(#<LOAD-OP> . #<CL-SOURCE-FILE "goodbye">)
......@@ -286,72 +271,68 @@ Note the lines marked with a \dag, which show that operations on composite compo
\end{alltt}
\end{minipage}
\centering
\caption{Sample \lisp{load-op} plan for the system definition in
Figure~\ref{fig:sampleASD}. We assume all files are previously compiled,
and \lisp{foo-utils} is already loaded.}
\caption{Sample {\loadOp} plan for the system definition in
Figure~\ref{fig:sampleASD}.
We assume that \lisp{foo-utils} is already compiled and loaded.}
\label{fig:samplePlan}
\end{figure}
\rtof{I killed the explanation of the distinction between in-order-to and
do-first. I first tried to rewrite this to explain the two --- because we
just jumped into talking about these things which don't appear in our example --- but then that
seemed to require me to explain how \defsystem{} expands \lisp{:depends-on}
into in-order-to and do-first, and pretty soon I felt I was digging myself further
and further into the weeds. If you think it's absolutely necessary to explain
these two, then we can have another try, but I'd rather not.}
% The \lisp{:depends-on} specifications in \defsystem{} are expanded into two
% kinds of dependencies:
% \lisp{in-order-to} dependencies, where changes ``upstream'' (in the depended-on components)
% will trigger a rebuild in the ``downstream'' components,
% and \lisp{do-first} dependencies that don't.
% For instance, consider some component \texttt{a}
% that \lisp{:depends-on} a component \texttt{b},
% both being the default component type a \lisp{cl-source-file}.
% Before \texttt{a} is either compiled (via operation \lisp{compile-op})
% or loaded (via operation \lisp{load-op}),
% \texttt{b} must first be loaded,
% which itself depends on \texttt{b} having been compiled;
% however, merely having to load \texttt{b} will not by itself
% force the (re)compilation of \texttt{a},
% but having to (re)compile \texttt{b} will
% force the (re)compilation of \texttt{a}.
% Therefore, {\traverse} first issues \lisp{in-order-to} dependencies,
% and {\opdonep} status, and
% if the operation needs to be done on the component,
% issues \lisp{do-first} dependencies;
% finally it issues the current operation and component pair
% as {\step}s of the plan.
\rtof{I think the following is a bit confusing. Do we intend to say that we
avoid forcing {\step}s that have \lisp{in-order-to} dependencies? Or is it
trying to say ``we force {\step}s that have \lisp{in-order-to} dependencies while
avoiding unnecessary {\step}s? Here's my attempt to rewrite...}
While building a plan, {\traverse} attempts to avoid including steps that are
unnecessary.
Steps may be \emph{forced} by changes in depended-on components, but if they are
not forced, they will be left out if \traverse{} can determine that they have
already been done.
To determine whether a {\step}
has already been done,
\traverse{} calls the generic function {\opdonep}.
For a compilation {\step}, \opdonep{} compares the timestamps of
the \lisp{input-files} and \lisp{output-files}
of the {\operation} and {\component}.
For a load {\step} with \lisp{input-files} but no \lisp{output-files},
it compares the timestamp of the input files
to the time it was last loaded into the current image (if ever).
Programmers extending the \ASDF{} protocol to new operations or components may
define their own methods on \opdonep{}.
%{\Step}s that are already up-to-date are skipped.
When building a plan, {\traverse} skips {\step}s
it can prove are not necessary.
A {\step} is necessary if it hasn't been done yet.
When a \emph{compilation} {\step} is necessary,
all the steps that depend on it are \emph{forced} to be necessary,
since whatever was done before will be out of date
by the time the {\step} is performed.
But if a {\step} and all the compilation {\step}s transitively
required in order to complete it
have already been done (by a previous run of {\ASDF}),
then the {\step} is not necessary and can be skipped.
To determine whether a {\step} has already been done,
{\traverse} calls the generic function {\opdonep}
on the {\operation} and {\component}.
For a compilation {\step},
wanted for its effects on the filesystem,
{\opdonep} compares the timestamps of
the corresponding \lisp{input-files} and \lisp{output-files}
(if present).
For a load {\step}, wanted for its effects on the current Lisp image,
there are no \lisp{output-files}, and
{\opdonep} compares the timestamp of the \lisp{input-files}
to the time they were last loaded into the current image (if ever).
Operations without either \lisp{input-files} nor \lisp{output-files}
(e.g., where the component is a {\module} or {\system})
are never considered necessary unless forced by their dependencies.
We will see subtleties of this part of the protocol,
when we discuss our modifications to {\traverse},
and the limitations on those modifications
(see Section \ref{sec:traverse-bug-fix}).
\hide{\footnote{
{\defsystem} internally expands
the specification of {\dependsOn} relationships between components
into two kinds of dependencies between related {\step}s:
{\inOrderTo} constraints that force {\step}s that depend on them,
and {\doFirst} constraints that do not.
Consider some component {\xa} that {\dependsOn} a component {\xb},
both being the default component type a \lisp{cl-source-file}.
Before {\xa} is either compiled (via operation {\compileOp})
or loaded (via operation {\loadOp}),
{\xb} must first be loaded in the current image,
which itself depends on {\xb} having been compiled.
However, merely having to load {\xb} will not by itself
force the (re)compilation of {\xa},
as compiling {\xa} from an equivalent set of loaded components
is expected to yield an equivalent result.
On the other hand, having to (re)compile {\xb} will
force the (re)compilation of {\xa}.
Therefore, {\traverse} first issues \lisp{in-order-to} dependencies,
and checks {\opdonep} status, and
if the operation needs to be done on the component,
recurses into {\doFirst} dependencies.
}}
%We will see subtleties of this part of the protocol,
%when we discuss our modifications to {\traverse},
%and the limitations on those modifications
%(see Section \ref{sec:traverse-bug-fix}).
% \fixme{There's a discussion of in-order-to on gmane, apparently.
% \url{http://article.gmane.org/gmane.lisp.cclan.general/674}. Check this
......@@ -372,18 +353,17 @@ but that has proven not to work,
because of the leakiness of the abstraction.
We discuss this further in Section \ref{sec:future-directions}.
Before we conclude the discussion, we would like to make two important
observations about the semantics of {\ASDF} system definitions.
These system definitions only specify
ordering dependencies between {\step}s.
The first observation is that, in general, these ordering dependencies only specify a partial order,
so for a given operation and system there may be multiple possible plans.
The second is that the semantics do not dictate a ``plan-then-execute'' implementation.
Indeed, a parallelizing extension to {\ASDF}, {\POIU},
Importantly, note that system definitions only specify
dependencies between {\step}s.
These dependencies only specify a partial order,
so for a given operation and system
there may be multiple possible complete plans.
Moreover, the semantics do not dictate
a ``plan-then-execute'' implementation.
And indeed, a parallelizing extension to {\ASDF}, {\POIU},
does things differently (see Section \ref{poiu}).
\hide{
\ftor{
If some points of semantics are too complex, vague or ambiguous
......@@ -423,7 +403,7 @@ does things differently (see Section \ref{poiu}).
plan --- they are the pieces out of which a plan is assembled (if {\make}
plans; I am not convinced this is the case). The parallel is the system definition in
{\ASDF}. That is not a plan; that is what the plan is assembled out of. For
example, if I ask {\ASDF} to do a load-op on a system, it will first build a
example, if I ask {\ASDF} to do a {\loadOp} on a system, it will first build a
plan, an ordered sequence of operation - component pairs. \emph{Then} it will
execute that plan. I don't believe that {\make} does anything like this. In
fact, I don't understand how make actually does its job. Similarly, I can
......
......@@ -65,6 +65,13 @@
\newcommand{\step}{step}
\newcommand{\Step}{Step}
\newcommand{\xa}{\texttt{a}}
\newcommand{\xb}{\texttt{b}}
\newcommand{\inOrderTo}{\lisp{in-order-to}}
\newcommand{\doFirst}{\lisp{do-first}}
\newcommand{\compileOp}{\lisp{compile-op}}
\newcommand{\loadOp}{\lisp{load-op}}
\newcommand{\dependsOn}{\lisp{:depends-on}}
\newcommand{\ftor}[1]{\draft{{\textbf{{\fare} to {\rpg}}: #1}}}
\newcommand{\rtof}[1]{\draft{{\textbf{{\rpg} to {\fare}}: #1}}}
......@@ -1460,7 +1467,7 @@ We found this to be harder than it first appeared.
%\input{history}
%\input{asdf}
\bibliographystyle{abbrv}
\bibliographystyle{abbrvurl}
\bibliography{asdf}
\end{document}
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment