Commit 0a93df92 authored by Francois-Rene Rideau's avatar Francois-Rene Rideau
Browse files

Just move the fixing traverse section after best practices.

parent 73ae10df
......@@ -684,181 +684,6 @@ in their input locations
without being foiled by translated output locations.
\section{Fixing {\Large \traverse}}
% \ftor{I still think we should only talk about bug fixes from the point of view of
% a general lesson to learned. Otherwise, the reader won't care.
% I say we move it any of it back into the section ``Best Practices'',
% though the subsection might be different from ``Backwards Compatibility''.
% For instance, the module bug might be either about
% backwards compatibility constraints in half-fixing bugs (the reason I put it there)
% or having simpler semantics, or having more understandable implementation,
% or testing more, etc. The \lisp{asdf:around} issue may be about the need for
% user extensibility, but also keeping things portable, or keeping interfaces
% simple.}
% There were also some implicit dependencies that seemed to be missing.
% For example (we fixed this one in {\ASDFii}),
% there was no implicit dependency indicating that
% in order to test (\class{test-op}) a \class{system}, one had to load it.
\label{sec:traverse-bug-fix} % if we move the traverse discussion, move this
% with it...
\begin{figure}[t]
\centering
\begin{minipage}{0.95\columnwidth}
\begin{verbatim}
(defsystem :test-module-depend
:components
((:file "file1")
(:module "quux"
:depends-on ("file1")
:components
((:file "file2")
(:module "file3mod"
:components
((:file "file3")))))))
\end{verbatim}
\end{minipage}
\caption{Example system illustrating the module dependency bug.}
\label{fig:moduleBugExample}
\end{figure}
One of the most annoying bugs we fixed for {\ASDFii}
was a problem with dependencies
involving composite components (modules and systems).
%% What are we compatible with and why?
%% See lp#479522
It illustrates both design issues in {\ASDF}
and limits imposed on us by backward compatibility in solving those issues.
However, ours was a partial fix, and we will have to tackle
a full solution in the future.
% Why introduce name ``composite'' instead of just telling that systems are modules?
In {\ASDFi}, the dependencies of a composite component
would fail to trigger recompilation of that component
and its dependents (the components that depend on it).
Figure \ref{fig:moduleBugExample} gives a system that shows the bug.
If one were to load this system, modify \file{file1.lisp},
and reload the system,
{\ASDFi} would fail to recompile \file{file2.lisp} or \file{file3.lisp}
the second time around.
The problem arises because composite components
are not like other, simple {\ASDF} components such as a {\clSourceFile}
% ``Never ascribe to decisions what can be ascribed to lack thereof.''
As per the original design of {\ASDF},
{\traverse} generates a plan made of
a sequence of elementary {\step}s to {\perform},
each a pair of {\operation} and {\component}.
This design can be contrasted with a maybe more intuitive model where
instead of a list of elementary {\step}s
the plan would have been a tree of
composite {\step}s and their constituent sub-{\step}s.
A consequence of this design is that the {\step}
corresponding to operating on the composite component
does not wrap {\em around} the {\step}s involving its constituents,
but is only a synchronization mark scheduled {\em after} all of them;
The {\perform} method over such an {\step} does nothing.
This foils the common desire of {\ASDF} system definers
to define {\around} methods for {\perform} that would e.g.
allow one to bind a dynamic variable (such as \lisp{*readtable*})
around all of the loading done to a module or system.
So, in {\ASDFi}, if one was to (compile and) load system
\lisp{test-module-depend}, touch \file{file2.lisp},
and then ask whether {\compileOp} had been done on module \lisp{quux}
(containing \file{file2}),
the answer would be \lisp{T}
(there are no effects to the {\perform} method,
there is nothing to do to re-do that,
and you should not force any dependency
because of that step not having been done yet),
although according to the maybe more intuitive model,
the answer would be ``no''
(there are effects required to compile the constituents of that module).
Trying to fix composite dependencies in {\traverse}, therefore,
opened a big can of worms.
{\ASDFi} originally contained special-purpose, \textit{ad hoc}, logic
for modules to decide when their components needed to be operated on
(since {\opdonep} could not be used), and this
special-purpose logic had some bugs.
Faced with buggy code we didn't understand,
we considered but quickly decided against
a radical reimplementation according
to the model we think was more intuitive,
that would give wrapping semantics to {\perform} on modules.
Indeed, we didn't dare make big sweeping changes because
we didn't understand the algorithm we were trying to fix,
and the consequences such fix would have on client systems
that may have depended on the existing model.
It is only after refactoring the whole implementation,
then later proofreading each other's explanations while writing this article,
that we finally understand the ins and outs of the algorithm.
As for the bug illustrated in Figure~\ref{fig:moduleBugExample}
(which was taken from the {\ASDFii} test suite),
we also decided to only fix it in the case of modules within a system,
but not in the case of systems:
{\ASDFii} still does not correctly handle the case where
a \emph{system} {\xa} that has changed that a system {\xb} depends on.
Some users had come to consider this as a {\em feature} rather than a bug,
and since we did not fully understand the issue then
and have strong opinions on the topic,
we were conservative and preserved this behavior.
The rationale of these users is that
unlike the files that internal modules depend on,
systems {\em usually} have stable interfaces,
and that when their interface changes,
either the client systems will change accordingly
(in which case {\ASDF} will detect that change and recompile them),
or things will otherwise break in obvious ways.
However, other users have complained that this is the wrong thing,
and that compilers are fast enough that
it is better to pay to recompile in the above cases,
to avoid wasting hours at debugging an invalid image
that wasn't properly recompiled
in the case that such a system interface changes in a subtly incompatible way
that causes an non-obvious bug (such as a by a modified macro definition).
We will probably do the right thing eventually,
and sacrifice backward compatibility with a dubious model
that we don't think anyone seriously relies upon,
now that we understand it.
See Section~\ref{sec:future-directions}.
% \rtof{The following discussion seems to me not so interesting. If you like, we
% could add a word about these issues in the section on portability.
% Portability has limited our use of some of the more arcane features of CLOS,
% and \emph{definitely} of the MOP. The former is not so bad as before, as the
% various {\CL} versions all now have much more complete CLOS implementations.
% What I suggest is that you either add
% something about this to the portability section, or not, and in any case you
% kill this rtof and the discussion below.}
% \rtof{
% Inelegance: bug-compatibility with early version of CLISP caused it
% to be written as a function when a generic function might have been cleaner.}
% \ftor{uh? What do you mean? I see no obvious trace of that in git log...}
% \rtof{This was documented by a comment in asdf.lisp way back when in the pre-git
% days. No method combination in CLISP in those days, so something that was
% thought of as possibly more elegantly written by an \lisp{append} or
% \lisp{progn} method combination was done more awkwardly. Not sure anyone
% cares about this, since \lisp{asdf::traverse} is now rewritten.}
% \ftor{As for the \lisp{asdf:around} method combination, it was used most importantly
% for \lisp{perform}, that already used some \lisp{asdf:around} internally
% to wrap the actual methods in some debugging restarts.
% In {\ASDFii} I worked around the issue by splitting it in two methods,
% \lisp{perform} that does the job and that the user may specialize with \lisp{:around},
% and \lisp{perform-with-restarts} that has the restart magic and calls \lisp{perform}
% }
\section{Engineering Best Practices}
\subsection{Sensible Data Structures}
......@@ -1082,6 +907,181 @@ Simple ideas can take literally hundreds of iterations to get right.
Documentation is a pain, and is still a weakness of {\ASDFii}.
\section{Fixing {\Large \traverse}}
% \ftor{I still think we should only talk about bug fixes from the point of view of
% a general lesson to learned. Otherwise, the reader won't care.
% I say we move it any of it back into the section ``Best Practices'',
% though the subsection might be different from ``Backwards Compatibility''.
% For instance, the module bug might be either about
% backwards compatibility constraints in half-fixing bugs (the reason I put it there)
% or having simpler semantics, or having more understandable implementation,
% or testing more, etc. The \lisp{asdf:around} issue may be about the need for
% user extensibility, but also keeping things portable, or keeping interfaces
% simple.}
% There were also some implicit dependencies that seemed to be missing.
% For example (we fixed this one in {\ASDFii}),
% there was no implicit dependency indicating that
% in order to test (\class{test-op}) a \class{system}, one had to load it.
\label{sec:traverse-bug-fix} % if we move the traverse discussion, move this
% with it...
\begin{figure}[t]
\centering
\begin{minipage}{0.95\columnwidth}
\begin{verbatim}
(defsystem :test-module-depend
:components
((:file "file1")
(:module "quux"
:depends-on ("file1")
:components
((:file "file2")
(:module "file3mod"
:components
((:file "file3")))))))
\end{verbatim}
\end{minipage}
\caption{Example system illustrating the module dependency bug.}
\label{fig:moduleBugExample}
\end{figure}
One of the most annoying bugs we fixed for {\ASDFii}
was a problem with dependencies
involving composite components (modules and systems).
%% What are we compatible with and why?
%% See lp#479522
It illustrates both design issues in {\ASDF}
and limits imposed on us by backward compatibility in solving those issues.
However, ours was a partial fix, and we will have to tackle
a full solution in the future.
% Why introduce name ``composite'' instead of just telling that systems are modules?
In {\ASDFi}, the dependencies of a composite component
would fail to trigger recompilation of that component
and its dependents (the components that depend on it).
Figure \ref{fig:moduleBugExample} gives a system that shows the bug.
If one were to load this system, modify \file{file1.lisp},
and reload the system,
{\ASDFi} would fail to recompile \file{file2.lisp} or \file{file3.lisp}
the second time around.
The problem arises because composite components
are not like other, simple {\ASDF} components such as a {\clSourceFile}
% ``Never ascribe to decisions what can be ascribed to lack thereof.''
As per the original design of {\ASDF},
{\traverse} generates a plan made of
a sequence of elementary {\step}s to {\perform},
each a pair of {\operation} and {\component}.
This design can be contrasted with a maybe more intuitive model where
instead of a list of elementary {\step}s
the plan would have been a tree of
composite {\step}s and their constituent sub-{\step}s.
A consequence of this design is that the {\step}
corresponding to operating on the composite component
does not wrap {\em around} the {\step}s involving its constituents,
but is only a synchronization mark scheduled {\em after} all of them;
The {\perform} method over such an {\step} does nothing.
This foils the common desire of {\ASDF} system definers
to define {\around} methods for {\perform} that would e.g.
allow one to bind a dynamic variable (such as \lisp{*readtable*})
around all of the loading done to a module or system.
So, in {\ASDFi}, if one was to (compile and) load system
\lisp{test-module-depend}, touch \file{file2.lisp},
and then ask whether {\compileOp} had been done on module \lisp{quux}
(containing \file{file2}),
the answer would be \lisp{T}
(there are no effects to the {\perform} method,
there is nothing to do to re-do that,
and you should not force any dependency
because of that step not having been done yet),
although according to the maybe more intuitive model,
the answer would be ``no''
(there are effects required to compile the constituents of that module).
Trying to fix composite dependencies in {\traverse}, therefore,
opened a big can of worms.
{\ASDFi} originally contained special-purpose, \textit{ad hoc}, logic
for modules to decide when their components needed to be operated on
(since {\opdonep} could not be used), and this
special-purpose logic had some bugs.
Faced with buggy code we didn't understand,
we considered but quickly decided against
a radical reimplementation according
to the model we think was more intuitive,
that would give wrapping semantics to {\perform} on modules.
Indeed, we didn't dare make big sweeping changes because
we didn't understand the algorithm we were trying to fix,
and the consequences such fix would have on client systems
that may have depended on the existing model.
It is only after refactoring the whole implementation,
then later proofreading each other's explanations while writing this article,
that we finally understand the ins and outs of the algorithm.
As for the bug illustrated in Figure~\ref{fig:moduleBugExample}
(which was taken from the {\ASDFii} test suite),
we also decided to only fix it in the case of modules within a system,
but not in the case of systems:
{\ASDFii} still does not correctly handle the case where
a \emph{system} {\xa} that has changed that a system {\xb} depends on.
Some users had come to consider this as a {\em feature} rather than a bug,
and since we did not fully understand the issue then
and have strong opinions on the topic,
we were conservative and preserved this behavior.
The rationale of these users is that
unlike the files that internal modules depend on,
systems {\em usually} have stable interfaces,
and that when their interface changes,
either the client systems will change accordingly
(in which case {\ASDF} will detect that change and recompile them),
or things will otherwise break in obvious ways.
However, other users have complained that this is the wrong thing,
and that compilers are fast enough that
it is better to pay to recompile in the above cases,
to avoid wasting hours at debugging an invalid image
that wasn't properly recompiled
in the case that such a system interface changes in a subtly incompatible way
that causes an non-obvious bug (such as a by a modified macro definition).
We will probably do the right thing eventually,
and sacrifice backward compatibility with a dubious model
that we don't think anyone seriously relies upon,
now that we understand it.
See Section~\ref{sec:future-directions}.
% \rtof{The following discussion seems to me not so interesting. If you like, we
% could add a word about these issues in the section on portability.
% Portability has limited our use of some of the more arcane features of CLOS,
% and \emph{definitely} of the MOP. The former is not so bad as before, as the
% various {\CL} versions all now have much more complete CLOS implementations.
% What I suggest is that you either add
% something about this to the portability section, or not, and in any case you
% kill this rtof and the discussion below.}
% \rtof{
% Inelegance: bug-compatibility with early version of CLISP caused it
% to be written as a function when a generic function might have been cleaner.}
% \ftor{uh? What do you mean? I see no obvious trace of that in git log...}
% \rtof{This was documented by a comment in asdf.lisp way back when in the pre-git
% days. No method combination in CLISP in those days, so something that was
% thought of as possibly more elegantly written by an \lisp{append} or
% \lisp{progn} method combination was done more awkwardly. Not sure anyone
% cares about this, since \lisp{asdf::traverse} is now rewritten.}
% \ftor{As for the \lisp{asdf:around} method combination, it was used most importantly
% for \lisp{perform}, that already used some \lisp{asdf:around} internally
% to wrap the actual methods in some debugging restarts.
% In {\ASDFii} I worked around the issue by splitting it in two methods,
% \lisp{perform} that does the job and that the user may specialize with \lisp{:around},
% and \lisp{perform-with-restarts} that has the restart magic and calls \lisp{perform}
% }
\section{Related work}
\label{sec:related-work-all}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment