Commit a644174b authored by Francois-Rene Rideau's avatar Francois-Rene Rideau
Browse files

My take on the traverse bug.

parent 63882ed4
......@@ -28,6 +28,7 @@
......@@ -39,7 +40,7 @@
......@@ -445,7 +446,7 @@ to modify libraries shared by others contribute to overall system reliability.
The mechanism for controlling output locations
is based on clever design by the original {\ASDF} author:
For \lisp{compile-op} operations on \lisp{cl-source-file} components,
For {\compileOp} operations on {\clSourceFile} components,
{\ASDF} will direct the {\CL} compiler to compile
the file pointed to by the \lisp{component-pathname} and write it to the
location pointed to by the \lisp{output-files} generic function.
......@@ -740,58 +741,44 @@ In {\ASDFi}, the dependencies of a composite component
would fail to trigger recompilation of that component
and its dependents (the components that depend on it).
Figure \ref{fig:moduleBugExample} gives a system that shows the bug.
If one were to load this system, then modify \file{file1.lisp},
that would \emph{not} cause \file{file2.lisp} or \file{file3.lisp}
to be recompiled when reloading system \lisp{test-module-depend}.
If one were to load this system, modify \file{file1.lisp},
and reload the system,
{\ASDFi} would fail to recompile \file{file2.lisp} or \file{file3.lisp}
the second time around.
The problem arises because composite components
are not like other, simple {\ASDF} component classes.
are not like other, simple {\ASDF} components such as a {\clSourceFile}
% ``Never ascribe to decisions what can be ascribed to lack thereof.''
In the original design of {\ASDF},
{\traverse} would generate a plan made of elementary steps,
each a pair of {\operation} and {\component},
that are to be {\perform}ed in sequence.
Whereas the {\traverse} method for a composite component recurses into
the constituents of the composite,
the step for composite component itself is scheduled to occur
after all its dependencies and constituents have been
suitably compiled and loaded;
performing that step does nothing,
scheduling it is just a way to
prevent recursing into it more than once.
This design did fit the original {\ASDF} goal of simplicity,
at least if by simple we mean that it uses sequences as chained cons cells
rather than implementing a new datastructure.
But it has turned out to have many unfortunate ramifications.
In general, it conflates the data of an operation and a component
as meaning either
``actions done to operate on a composite that are not actions on a component''
(let us refer to these as ``synergistic actions'') % what does that mean???
and ``the action of operating on a composite'' in ways that are unfortunate.
% From there to end of section, I'm lost.
More specifically, because these synergistic actions are typically to be done
whenever a system is loaded, the {\opdonep} method for arbitrary
operations, would not work properly to indicate whether the operation had been
done for that composite, but only to indicate whether the
\emph{synergistic actions} had been performed.
As per the original design of {\ASDF},
{\traverse} generates a plan made of
a sequence of elementary {\step}s to {\perform},
each a pair of {\operation} and {\component}.
This design can be contrasted with a maybe more intuitive model where
instead of a list of elementary {\step}s
the plan would have been a tree of
composite {\step}s and their constituent sub-{\step}s.
A consequence of this design is that the {\step}
corresponding to operating on the composite component
does not wrap {\em around} the {\step}s involving its constituents,
but is only a synchronization mark scheduled {\em after} all of them;
The {\perform} method over such an {\step} does nothing.
This foils the common desire of {\ASDF} system definers
to define {\around} methods for {\perform} that would e.g.
allow one to bind a dynamic variable (such as \lisp{*readtable*})
around all of the loading done to a module or system.
So, in {\ASDFi}, if one was to (compile and) load system
\lisp{test-module-depend}, touch \file{file2.lisp}, and then ask whether
\lisp{compile-op} had been done on module \lisp{quux} (containing \file{file2}),
the answer would be \lisp{T}, although intuitively the answer should be ``no.''
Similarly, a plan for loading \lisp{:test-module-depend} would contain an
instruction to load \file{file2}, to load \file{file3}, and then later to load
\lisp{file3mod}, to load \lisp{quux}, and finally to load
For this reason, it was not possible to modulate the loading of all of
\lisp{quux} by defining an \lisp{:around} method on {\perform} as applied to
\lisp{load-op} and the module \lisp{quux}. A common desire in {\ASDF} system
definers is to do something like bind a dynamic variable (say \lisp{*readtable*}) around all of the
loading done to a module like \lisp{quux}. Unfortunately doing this in the
{\perform} method on \lisp{quux} will not have the desired effect, because that
{\perform} is done \emph{after} the operation on the components, not
\emph{around} the operations on the components.
\lisp{test-module-depend}, touch \file{file2.lisp},
and then ask whether {\compileOp} had been done on module \lisp{quux}
(containing \file{file2}),
the answer would be \lisp{T}
(there are no effects to the {\perform} method,
there is nothing to do to re-do that,
and you should not force any dependency
because of that step not having been done yet),
although according to the maybe more intuitive model,
the answer would be ``no''
(there are effects required to compile the constituents of that module).
Trying to fix composite dependencies in {\traverse}, therefore,
opened a big can of worms.
......@@ -799,31 +786,48 @@ opened a big can of worms.
for modules to decide when their components needed to be operated on
(since {\opdonep} could not be used), and this
special-purpose logic had some bugs.
Instead of radically re-engineering the logic, we simply repaired it in place
(and later {\fare} refactored it), fixing the bug illustrated in
(which was taken from the {\ASDFii} test suite).
We determined that we could not change the semantics of performing operations on
composites from synergistic actions to composite actions, without danger of
breaking an arbitrary amount of existing systems.
Unfortunately, we were limited to fixing the dependency bug for the module
case: {\ASDFii} still does not correctly handle the case where an upstream
\emph{system} has changed. Here the problem is that we do not have clear
semantics for system changes that are sufficient to detect the need to recompile
both across and within {\CL} sessions. See Section~\ref{sec:future-directions}, where we
discuss some more radical solutions that should still maintain some forms of
backward compatibility.
% \rtof{As I wrote this, I recalled that this illustrates cases where the poor
% {\ASDF} user really \emph{does} need to understand internals of the system.
% For example, the user really does need to understand the odd semantics of
% composite operations in order to know why his \lisp{around} methods don't do
% what he wants... Should we back patch that or just not worry about it. I
% suppose this enters into the question of where to put this material.}
% \hrulefill{}
Faced with buggy code we didn't understand,
we considered but quickly decided against
a radical reimplementation according
to the model we think was more intuitive,
that would give wrapping semantics to {\perform} on modules.
Indeed, we didn't dare make big sweeping changes because
we didn't understand the algorithm we were trying to fix,
and the consequences such fix would have on client systems
that may have depended on the existing model.
It is only after refactoring the whole implementation,
then later proofreading each other's explanations while writing this article,
that we finally understand the ins and outs of the algorithm.
As for the bug illustrated in Figure~\ref{fig:moduleBugExample}
(which was taken from the {\ASDFii} test suite),
we also decided to only fix it in the case of modules within a system,
but not in the case of systems:
{\ASDFii} still does not correctly handle the case where
a \emph{system} {\xa} that has changed that a system {\xb} depends on.
Some users had come to consider this as a {\em feature} rather than a bug,
and since we did not fully understand the issue then
and have strong opinions on the topic,
we were conservative and preserved this behavior.
The rationale of these users is that
unlike the files that internal modules depend on,
systems {\em usually} have stable interfaces,
and that when their interface changes,
either the client systems will change accordingly
(in which case {\ASDF} will detect that change and recompile them),
or things will otherwise break in obvious ways.
However, other users have complained that this is the wrong thing,
and that compilers are fast enough that
it is better to pay to recompile in the above cases,
to avoid wasting hours at debugging an invalid image
that wasn't properly recompiled
in the case that such a system interface changes in a subtly incompatible way
that causes an non-obvious bug (such as a by a modified macro definition).
We will probably do the right thing eventually,
and sacrifice backward compatibility with a dubious model
that we don't think anyone seriously relies upon,
now that we understand it.
See Section~\ref{sec:future-directions}.
% \rtof{The following discussion seems to me not so interesting. If you like, we
% could add a word about these issues in the section on portability.
......@@ -854,11 +858,6 @@ backward compatibility.
% and \lisp{perform-with-restarts} that has the restart magic and calls \lisp{perform}
% }
% \rtof{the \lisp{asdf:around} issue was a different one. There were early
% comments saying that method combination might have made things better --- I
% believe people referred to possible uses of \lisp{progn} and \lisp{append}
% method combination. But I believe we could leave those out.}
\section{Engineering Best Practices}
......@@ -988,10 +987,14 @@ may have been independently specified.
We also had to either rebind or shadow symbols for redefined functions
depending on implementation as discussed in Section \ref{sec:upgradeability}.
Additionally, in order to eliminate all warnings on all implementations,
we also had to be careful to
we had to be careful to
include \lisp{ignorable} declarations in \lisp{defmethod}s, and
avoid forward references to yet undefined and undeclared functions.
We also have several portable wrappers over implementation-specific functions
We also had to revert the previous use of the long form of
\lisp{define-method-combination}, that some implementations did not support,
instead introducing additional generic functions as extension points
for which users may define methods.
Finally, we wrote several portable wrappers over implementation-specific functions
to access the environment, such as \lisp{getenv}, and
tens of small implementation-specific adaptations.
......@@ -1038,7 +1041,7 @@ Similarly, on Windows, support for shortcuts had to go into the core.
With all the things we added,
{\ASDF} almost doubled in size since we started working on {\ASDFii},
and almost quadrupled since the original author left.\footnote{
Release 2.006 is 144996 byte long;
Our current release 2.007 is 146000 byte long;
1.369, the last release by Gary King was 77079 byte long;
the last version by Daniel Barlow, in 2004, was 38881 byte long.
......@@ -1065,8 +1068,8 @@ First, it was very impressive that
{\POIU} could be written
without cooperation from the {\ASDF} maintainers.
This was achieved
thanks to CLOS class redefinition.
Second, the power of CLOS
thanks to {\CLOS} class redefinition.
Second, the power of {\CLOS}
allowed us to keep the interface between {\POIU} and {\ASDF}
very simple and informal.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment