Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
asdf
ilc2010
Commits
63882ed4
Commit
63882ed4
authored
Aug 26, 2010
by
Francois-Rene Rideau
Browse files
Another pass at asdf plan generation
parent
7dbc8dd8
Changes
2
Hide whitespace changes
Inline
Side-by-side
asdf.tex
View file @
63882ed4
...
...
@@ -40,28 +40,13 @@ To that end, it will take the following steps:
(3) execute this plan.
%\end{itemize}
In addition to loading others' systems, system builders will write
their own system
definition
s.
Most developers load and modify existing systems.
Advanced developers define
their own systems.
More advanced developers extend
{
\ASDF
}
's object model:
they may define new
{
\em
components
}
in addition to Lisp source files,
such as C source files or protocol buffer definitions;
or they may define new
{
\em
operations
}
in addition to compiling and loading,
such as documentation generation or regression testing.
% Unfortunately, {\ASDF} has oddities that have, all too often,
% hindered their ability to use {\ASDF} as a tool.
% I cut all of the following --- it's redundant because in the next section we
% explain that ASDF does more than make does, in just the ways we explain here.
% {\ASDF} provides software management for {\CL}.
% \rtof{I'm sorry to be a nitpicker, but I know we've gone around on this before.
% ```Notoriously'' is just the wrong word here. Here's the wordnet definition:
% ``ill-famed, infamous, notorious (known widely and usually unfavorably) "a
% notorious gangster"; "the tenderloin district was notorious for vice"; "the
% infamous Benedict Arnold";''}
% Not only can it be used to build {\CL} software;
% it can also help with a wider range of software management tasks,
% such as testing, documentation or packaging.
{
\ASDF
}
is a central piece of software for the
{
\CL
}
community:
{
\CL
}
libraries --- and especially the open source libraries ---
...
...
@@ -84,21 +69,20 @@ they otherwise differ in their goals, their design,
the constraints they respect, their internal architecture,
the concepts that underly them, the interfaces they expose to users.
\fixme
{
We use ``analogous'' too many times in the following paragraph.
}
{
\ASDF
}
finds systems and loads systems,
problems that are not handled
by build tools such as
{
\make
}
or
{
\ant
}
.
These other build tools don't search for systems;
they
must be pointed at a system definition
These other build tools don't search for systems;
they
must be pointed at a system definition
in the current or specified directory.
Finding systems at build time would be analogous
to a subset of
\texttt
{
libtool
}
;
finding and loading systems at runtime would
be more analogous
to
a subset of the Unix dynamic linker
\texttt
{
ld.so
}
.
finding and loading systems at runtime would
correspond to
a subset of the Unix dynamic linker
\texttt
{
ld.so
}
.
As for loading,
the fact that
{
\ASDF
}
is available for interactive use
makes it analogous to some component of the operating system shell;
loading some systems might thus be
analogous
to
loading some systems might thus be
similar
to
importing shell functions, starting a daemon,
registering a plugin in your browser,
loading code into some master process, etc.
...
...
@@ -111,13 +95,15 @@ to modify previously-loaded systems in order to accommodate those changes.
{
\ASDF
}
also differs from
{
\make
}
in terms of how systems are specified.
{
\make
}
is built around a complex combination of
multiple layers of languages --- some domain-specific languages and some
generalized programming languages --- used in
\emph
{
makefiles
}
.
multiple layers of languages --- some domain-specific languages and
some generalized programming languages.
{
\make
}
interprets a
\emph
{
makefile
}
thusly:
% \rtof{I would like to cut the following; I think it's enough that the reader
% knows that there are multiple languages involved.}
% a text substitution program is expanded into
% simple pattern matching rules used by an inference engine
% that will run simply parameterized shell scripts.
% \ftor{I think it brings some insight as to the nature of the difference}
a text substitution program is expanded into
simple pattern matching rules used by an inference engine
that will run simply parameterized shell scripts.
{
\make
}
is a powerful tool that can express arbitrary programs,
but makefiles can be a mesh of code that defies any simple analysis.
{
\ASDF
}
is a small
{
\CL
}
program,
...
...
@@ -214,16 +200,20 @@ Systems can be recursively organized in a tree of {\module}s
that may or may not map to a similar tree of directories.
System management tasks consist in applying the generic function
{
\operate
}
on parameters specifying an
{
\operation
}
and a
{
\system
}
,
the latter of which may be designated by a string or a symbol.
{
\operate
}
will first find and load the proper system definition, if necessary.
{
\operate
}
on parameters specifying an
{
\operation
}
and a
{
\system
}
.
The
{
\operation
}
is either a
{
\loadOp
}
,
specifying that a system or component is to be loaded into the current image,
or a
{
\compileOp
}
specifying that a system or component is to be compiled into the filesystem.
The
{
\system
}
is designated by a string or a symbol, and
{
\operate
}
will first find and load the thus named system definition.
This is done by the generic function
\lisp
{
find-system
}
, which searches
for the system definition in a configurable system registry
(and its in-memory cache).
The system definition search is one of the aspects of
{
\ASDFi
}
that we reformed;
the previous protocol had to be initialized using arbitrary Lisp code,
and had a number of undesirable features.
S
ee Section
\ref
{
sec:input-locations
}
.
%
the previous protocol had to be initialized using arbitrary Lisp code,
%
and had a number of undesirable features.
s
ee Section
\ref
{
sec:input-locations
}
.
After a system definition is found,
{
\operate
}
generates a
\emph
{
plan
}
for completing the
{
\operation
}
...
...
@@ -242,32 +232,27 @@ making it easy to fix simple mistakes without interrupting the operation.
\subsection
{
Plan generation
}
\label
{
sec:plan-generation
}
\rtof
{
I moved the material that was in your first paragraph here down to the end
of the section.
That material was all
\emph
{
correct
}
and good to have, but putting it in front
amounted to presenting all of the exceptions and hedges first, before the
big picture explanation.
}
{
\traverse
}
performs a depth-first, postorder traversal
The plan-generating function
{
\traverse
}
performs a depth-first, postorder traversal
over the
{
\step
}
s needed to operate on the target system and its dependencies.
Each
{
\step
}
is a pair of an
{
\operation
}
and a
{
\component
}
.
Each
{
\step
}
is a pair of an
{
\operation
}
and a
{
\component
}
,
What we mean by postorder here is that
when applying an
{
\operation
}
to a
{
\system
}
(or
{
\module
}
),
the plan will contain
{
\step
}
s
first to complete the
{
\operation
}
to the sub-components,
then to perform it on the
{
\system
}
(or
{
\module
}
) itself. For example, a load
op plan for the system in Figure~
\ref
{
fig:sampleASD
}
is given as
Figure~
\ref
{
fig:samplePlan
}
.
Note the lines marked with a
\dag
, which show that operations on composite components
(
\module
{}
s and
\system
{}
s) are scheduled after operations on their components.
then to perform it on the
{
\system
}
(or
{
\module
}
) itself.
For example, a
{
\loadOp
}
plan
for the system in Figure~
\ref
{
fig:sampleASD
}
is given as Figure~
\ref
{
fig:samplePlan
}
.
Note the lines marked with a
\dag
,
which show that operations on composite components (
{
\module
}
s and
{
\system
}
s)
are scheduled after operations on their components.
\begin{figure}
[t]
\begin{minipage}
{
1.0
\columnwidth
}
\begin{alltt}
((#<COMPILE-OP> . #<CL-SOURCE-FILE "packages">)
((#<COMPILE-OP> . #<CL-SOURCE-FILE "packages">)
(#<LOAD-OP> . #<CL-SOURCE-FILE "packages">)
(#<COMPILE-OP> . #<CL-SOURCE-FILE "macros">)
(#<COMPILE-OP> . #<CL-SOURCE-FILE "classes">)
...
...
@@ -277,7 +262,7 @@ Note the lines marked with a \dag, which show that operations on composite compo
(#<COMPILE-OP> . #<CL-SOURCE-FILE "hello">)
(#<LOAD-OP> . #<CL-SOURCE-FILE "hello">)
(#<COMPILE-OP> . #<CL-SOURCE-FILE "goodbye">)
\dag
(#<COMPILE-OP> . #<MODULE "main">)
\dag
(#<COMPILE-OP> . #<MODULE "main">)
(#<COMPILE-OP> . #<SYSTEM "hello-lisp">)
(#<LOAD-OP> . #<CL-SOURCE-FILE "methods">)
(#<LOAD-OP> . #<CL-SOURCE-FILE "goodbye">)
...
...
@@ -286,72 +271,68 @@ Note the lines marked with a \dag, which show that operations on composite compo
\end{alltt}
\end{minipage}
\centering
\caption
{
Sample
\
lisp
{
load
-o
p
}
plan for the system definition in
Figure~
\ref
{
fig:sampleASD
}
.
We assume all files are previously compiled,
and
\lisp
{
foo-utils
}
is already loaded.
}
\caption
{
Sample
{
\load
O
p
}
plan for the system definition in
Figure~
\ref
{
fig:sampleASD
}
.
We assume that
\lisp
{
foo-utils
}
is already
compiled and
loaded.
}
\label
{
fig:samplePlan
}
\end{figure}
\rtof
{
I killed the explanation of the distinction between in-order-to and
do-first. I first tried to rewrite this to explain the two --- because we
just jumped into talking about these things which don't appear in our example --- but then that
seemed to require me to explain how
\defsystem
{}
expands
\lisp
{
:depends-on
}
into in-order-to and do-first, and pretty soon I felt I was digging myself further
and further into the weeds. If you think it's absolutely necessary to explain
these two, then we can have another try, but I'd rather not.
}
% The \lisp{:depends-on} specifications in \defsystem{} are expanded into two
% kinds of dependencies:
% \lisp{in-order-to} dependencies, where changes ``upstream'' (in the depended-on components)
% will trigger a rebuild in the ``downstream'' components,
% and \lisp{do-first} dependencies that don't.
% For instance, consider some component \texttt{a}
% that \lisp{:depends-on} a component \texttt{b},
% both being the default component type a \lisp{cl-source-file}.
% Before \texttt{a} is either compiled (via operation \lisp{compile-op})
% or loaded (via operation \lisp{load-op}),
% \texttt{b} must first be loaded,
% which itself depends on \texttt{b} having been compiled;
% however, merely having to load \texttt{b} will not by itself
% force the (re)compilation of \texttt{a},
% but having to (re)compile \texttt{b} will
% force the (re)compilation of \texttt{a}.
% Therefore, {\traverse} first issues \lisp{in-order-to} dependencies,
% and {\opdonep} status, and
% if the operation needs to be done on the component,
% issues \lisp{do-first} dependencies;
% finally it issues the current operation and component pair
% as {\step}s of the plan.
\rtof
{
I think the following is a bit confusing. Do we intend to say that we
avoid forcing
{
\step
}
s that have
\lisp
{
in-order-to
}
dependencies? Or is it
trying to say ``we force
{
\step
}
s that have
\lisp
{
in-order-to
}
dependencies while
avoiding unnecessary
{
\step
}
s? Here's my attempt to rewrite...
}
While building a plan,
{
\traverse
}
attempts to avoid including steps that are
unnecessary.
Steps may be
\emph
{
forced
}
by changes in depended-on components, but if they are
not forced, they will be left out if
\traverse
{}
can determine that they have
already been done.
To determine whether a
{
\step
}
has already been done,
\traverse
{}
calls the generic function
{
\opdonep
}
.
For a compilation
{
\step
}
,
\opdonep
{}
compares the timestamps of
the
\lisp
{
input-files
}
and
\lisp
{
output-files
}
of the
{
\operation
}
and
{
\component
}
.
For a load
{
\step
}
with
\lisp
{
input-files
}
but no
\lisp
{
output-files
}
,
it compares the timestamp of the input files
to the time it was last loaded into the current image (if ever).
Programmers extending the
\ASDF
{}
protocol to new operations or components may
define their own methods on
\opdonep
{}
.
%{\Step}s that are already up-to-date are skipped.
When building a plan,
{
\traverse
}
skips
{
\step
}
s
it can prove are not necessary.
A
{
\step
}
is necessary if it hasn't been done yet.
When a
\emph
{
compilation
}
{
\step
}
is necessary,
all the steps that depend on it are
\emph
{
forced
}
to be necessary,
since whatever was done before will be out of date
by the time the
{
\step
}
is performed.
But if a
{
\step
}
and all the compilation
{
\step
}
s transitively
required in order to complete it
have already been done (by a previous run of
{
\ASDF
}
),
then the
{
\step
}
is not necessary and can be skipped.
To determine whether a
{
\step
}
has already been done,
{
\traverse
}
calls the generic function
{
\opdonep
}
on the
{
\operation
}
and
{
\component
}
.
For a compilation
{
\step
}
,
wanted for its effects on the filesystem,
{
\opdonep
}
compares the timestamps of
the corresponding
\lisp
{
input-files
}
and
\lisp
{
output-files
}
(if present).
For a load
{
\step
}
, wanted for its effects on the current Lisp image,
there are no
\lisp
{
output-files
}
, and
{
\opdonep
}
compares the timestamp of the
\lisp
{
input-files
}
to the time they were last loaded into the current image (if ever).
Operations without either
\lisp
{
input-files
}
nor
\lisp
{
output-files
}
(e.g., where the component is a
{
\module
}
or
{
\system
}
)
are never considered necessary unless forced by their dependencies.
We will see subtleties of this part of the protocol,
when we discuss our modifications to
{
\traverse
}
,
and the limitations on those modifications
(see Section
\ref
{
sec:traverse-bug-fix
}
).
\hide
{
\footnote
{
{
\defsystem
}
internally expands
the specification of
{
\dependsOn
}
relationships between components
into two kinds of dependencies between related
{
\step
}
s:
{
\inOrderTo
}
constraints that force
{
\step
}
s that depend on them,
and
{
\doFirst
}
constraints that do not.
Consider some component
{
\xa
}
that
{
\dependsOn
}
a component
{
\xb
}
,
both being the default component type a
\lisp
{
cl-source-file
}
.
Before
{
\xa
}
is either compiled (via operation
{
\compileOp
}
)
or loaded (via operation
{
\loadOp
}
),
{
\xb
}
must first be loaded in the current image,
which itself depends on
{
\xb
}
having been compiled.
However, merely having to load
{
\xb
}
will not by itself
force the (re)compilation of
{
\xa
}
,
as compiling
{
\xa
}
from an equivalent set of loaded components
is expected to yield an equivalent result.
On the other hand, having to (re)compile
{
\xb
}
will
force the (re)compilation of
{
\xa
}
.
Therefore,
{
\traverse
}
first issues
\lisp
{
in-order-to
}
dependencies,
and checks
{
\opdonep
}
status, and
if the operation needs to be done on the component,
recurses into
{
\doFirst
}
dependencies.
}}
%We will see subtleties of this part of the protocol,
%when we discuss our modifications to {\traverse},
%and the limitations on those modifications
%(see Section \ref{sec:traverse-bug-fix}).
% \fixme{There's a discussion of in-order-to on gmane, apparently.
% \url{http://article.gmane.org/gmane.lisp.cclan.general/674}. Check this
...
...
@@ -372,18 +353,17 @@ but that has proven not to work,
because of the leakiness of the abstraction.
We discuss this further in Section
\ref
{
sec:future-directions
}
.
Before we conclude the discussion, we would like to make two important
observations about the semantics of
{
\ASDF
}
system definition
s.
These
system definitions only specify
ordering dependencies between
{
\step
}
s.
T
he
first observation is that, in general, these ordering dependencies only specify a partial order,
so for a given operation and system there may be multiple possible plans.
The second is that the semantics do not dictate
a ``plan-then-execute'' implementation.
I
ndeed, a parallelizing extension to
{
\ASDF
}
,
{
\POIU
}
,
Importantly, note that system definitions only specify
dependencies between
{
\step
}
s.
These
dependencies only specify a partial order,
so for a given operation and system
t
he
re may be multiple possible complete plans.
Moreover, the semantics do not dictate
a ``plan-then-execute'' implementation.
And i
ndeed, a parallelizing extension to
{
\ASDF
}
,
{
\POIU
}
,
does things differently (see Section
\ref
{
poiu
}
).
\hide
{
\ftor
{
If some points of semantics are too complex, vague or ambiguous
...
...
@@ -423,7 +403,7 @@ does things differently (see Section \ref{poiu}).
plan --- they are the pieces out of which a plan is assembled (if
{
\make
}
plans; I am not convinced this is the case). The parallel is the system definition in
{
\ASDF
}
. That is not a plan; that is what the plan is assembled out of. For
example, if I ask
{
\ASDF
}
to do a load
-op
on a system, it will first build a
example, if I ask
{
\ASDF
}
to do a
{
\
load
Op
}
on a system, it will first build a
plan, an ordered sequence of operation - component pairs.
\emph
{
Then
}
it will
execute that plan. I don't believe that
{
\make
}
does anything like this. In
fact, I don't understand how make actually does its job. Similarly, I can
...
...
main.tex
View file @
63882ed4
...
...
@@ -65,6 +65,13 @@
\newcommand
{
\step
}{
step
}
\newcommand
{
\Step
}{
Step
}
\newcommand
{
\xa
}{
\texttt
{
a
}}
\newcommand
{
\xb
}{
\texttt
{
b
}}
\newcommand
{
\inOrderTo
}{
\lisp
{
in-order-to
}}
\newcommand
{
\doFirst
}{
\lisp
{
do-first
}}
\newcommand
{
\compileOp
}{
\lisp
{
compile-op
}}
\newcommand
{
\loadOp
}{
\lisp
{
load-op
}}
\newcommand
{
\dependsOn
}{
\lisp
{
:depends-on
}}
\newcommand
{
\ftor
}
[1]
{
\draft
{{
\textbf
{{
\fare
}
to
{
\rpg
}}
: #1
}}}
\newcommand
{
\rtof
}
[1]
{
\draft
{{
\textbf
{{
\rpg
}
to
{
\fare
}}
: #1
}}}
...
...
@@ -1460,7 +1467,7 @@ We found this to be harder than it first appeared.
%\input{history}
%\input{asdf}
\bibliographystyle
{
abbrv
}
\bibliographystyle
{
abbrv
url
}
\bibliography
{
asdf
}
\end{document}
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment