Commit 55bc27f2 authored by Francois-Rene Rideau's avatar Francois-Rene Rideau

Update TODO wrt support for proper phase separation

parent 1ad4042c
* Decree that versions older than 2 years are not supported;
yet, keep supporting an upgrade from whatever vendors provide
— so push them to provide recent stuff.
yet, keep supporting an upgrade from whatever vendors provide
— so push them to provide recent stuff.
** Already, swank-asdf doesn't support anything older than 2.014.6
(as provided by the first 2011 iteration of Quicklisp), and
I (fare) am pushing for swank-asdf to not support anything older
......@@ -17,7 +17,13 @@
** Make tests stateless and/or make any modifications private,
so that they can be run in parallel.
* Moving toward cross-compilation (ASDF 3.2? 4?)
* Testing support
** Replace test-op with run-test-op and test-report-op ?
* Implement a style-checker, that will issue STYLE-WARNINGs
when you don't follow all the recommended guidelines.
* Moving toward cross-compilation (ASDF 3.4? 4?)
This would allow to get most of the benefits of XCVB
(reproducibility, scalability, applicability to mobile platforms, etc.),
except without the backward incompatibility of XCVB.
......@@ -112,14 +118,14 @@
make t l=cmucl t=test-encodings.script
which was a different PCL bug from the above (without same workaround)
but possibly also triggered by the non-standard declaim.
*** `#5(1 ,@`(2 3)))` returns #(1 2 3),
rather than #(1 2 3 2 3 2 3 2 3) or even better #(1 2 3 3 3).
*** `#5(1 ,@\`(2 3)))` returns `#(1 2 3)`,
rather than `#(1 2 3 2 3 2 3 2 3)` or even better `#(1 2 3 3 3)`.
*** It purports to support Unicode, but it has only 16-bit characters.
Note that (subtypep 'character 'base-char) says T -- everything is a base char(!)
** SCL has bugs:
*** it doesn't like run-program from a modified directory?
*** it somehow pushes :non-base-chars-exist-p even though +non-base-chars-exist-p+ is NIL???
*** it somehow pushes `:non-base-chars-exist-p` even though `+non-base-chars-exist-p+` is `NIL`???
** XCL has bad bugs:
*** make-pathname doesn't handle :type nil properly and
......@@ -140,6 +146,7 @@
(make-pathname :name "foo" :type :unspecific) ==> #P"foo."
If bug is ever fixed, upgrade *unspecific-pathname-type* in uiop/pathname.
*** XCL doesn't properly process files the name of which contains a *
*** XCL fails to recognize that specialized method parameters are implicitly ignorable.
** GCL is almost working again; but implementation bugs remain.
See Francois-Rene Rideau's messages on gcl-devel starting November 2013.
......@@ -154,33 +161,33 @@
INTERNAL-SIMPLE-ERROR: The package named ASDF/INTERFACE, does not exist
*** Another GCL compiler bug:
when I changed the definition of getcwd from
(let ((*default-pathname-defaults* #p"")) (truename #p"")) to
(let ((*default-pathname-defaults* *nil-pathname*)) (truename *nil-pathname*))
`(let ((*default-pathname-defaults* #p"")) (truename #p""))` to
`(let ((*default-pathname-defaults* *nil-pathname*)) (truename *nil-pathname*))`
to guard against e.g. a logical-pathname context while loading asdf
and parsing #p"", calls to getcwd result in a segfault.
and parsing `#p""`, calls to `getcwd` result in a segfault.
*** An another bug: gcl refuses dynamic-extent declaration on functions.
uiop/stream.lisp: #-gcl (declare (dynamic-extent ,@(when
before `(#',beforef)) ,@(when after `(#',afterf))))
*** (typep p 'logical-pathname) should be T if p has a logical-pathname host.
*** apropos is case-sensitive and returns a same symbol many times
```uiop/stream.lisp: #-gcl (declare (dynamic-extent ,@(when
before `(#',beforef)) ,@(when after `(#',afterf))))```
*** `(typep p 'logical-pathname)` should be T if p has a logical-pathname host.
*** `apropos` is case-sensitive and returns a same symbol many times
(presumably once per package where it is present,
instead of just once for its home package)
*** compile-file fails to return proper secondary values in case of non-style WARNING.
*** (pathname-directory #p"foo/") is incorrectly ("foo") instead of (:RELATIVE "foo")
*** `compile-file` fails to return proper secondary values in case of non-style `WARNING`.
*** `(pathname-directory #p"foo/")` is incorrectly `("foo")` instead of `(:RELATIVE "foo")`
*** Do whatever it takes to pass the asdf tests, add the above?
*** Trying to uiop:slurp-stream-forms from a stream with #+(or) :foo
(or read-file-forms from an file with same) results in an error,
rather than nil. This is probably a bug in #+ processing.
*** Trying to `uiop:slurp-stream-forms` from a stream with `#+(or) :foo`
(or `read-file-forms` from an file with same) results in an error,
rather than `nil`. This is probably a bug in `#+` processing.
Unhappily, debian creates such a file in
/etc/common-lisp/asdf-output-translations.conf.d/01-common-lisp-controller.conf
`/etc/common-lisp/asdf-output-translations.conf.d/01-common-lisp-controller.conf`
*** Tests that try to catch an error fail (but catching a warning succeeds),
which suggests brokenness in handler-bind and/or error.
*** COMPILE-FILE* fails due to handler-bind error in
with-muffled-compiler-conditions or so.
*** `#5(1 ,@`(2 3)))` returns #(1 2 3),
rather than #(1 2 3 2 3 2 3 2 3) or even better #(1 2 3 3 3).
*** (DIRECTORY #p"*.*") fails to match files with pathname type NIL.
*** GCL doesn't properly process files the name of which contains a *
*** `COMPILE-FILE*` fails due to handler-bind error in
`with-muffled-compiler-conditions` or so.
*** `#5(1 ,@`(2 3)))` returns `#(1 2 3)`,
rather than `#(1 2 3 2 3 2 3 2 3)` or even better `#(1 2 3 3 3)`.
*** `(DIRECTORY #p"*.*")` fails to match files with pathname-type `NIL`.
*** GCL doesn't properly process files the name of which contains a `*`
** ABCL has a few bugs.
*** ABCL fails the timestamp propagation test.
......@@ -243,12 +250,6 @@
** Real solution: defer parsing and evaluation of defsystem forms?
Or simply detect circularity and issue an appropriate error?
* Implement proper incrementality for defsystem-depends-on.
** For proper incremental redefinitions, systems probably need to
depend-on (load-asd-op . primary-system) which itself will
depend-on all the load-op of defsystem-depends-on in that file.
https://bugs.launchpad.net/asdf/+bug/1500578
* Include some ABL test for stassats's (now obsolete?) thing:
(asdf:enable-asdf-binary-locations-compatibility
:centralize-lisp-binaries t :default-toplevel-directory *fasl-dir*)
......@@ -382,6 +383,20 @@ It looks like SWANK can be fixed soon, though, so we'll see.
version
* ASDF4: search for this tag, rename things (incompatibly, thus) and cleanup code.
** Migrate from component-depends-on to action-depends-on
*** I contend a future version of ASDF will replace
`(component-depends-on operation component)`
with `(action-depends-on plan operation component)`.
This allows for different normalization strategies for dependencies
(including strategies that avoid resolving things to NIL),
a reified context for featurep checks, etc.
*** Easy but long backward-compatible transition:
**** get all users to use the new gf and keep extending both new and old gf,
meanwhile the new gf has an around method that actually calls the old gf
except in testing mode and/or if we can check that they're using the new convention
**** when everyone has migrated, remove the old mode and the short-circuit.
*** However, we cannot deprecate component-depends-on yet — not until we have
some transition in place to a better interface.
* Documentation
** See message from 2014-01-27 05:26:44 GMT for stuff to document.
......@@ -417,22 +432,6 @@ It looks like SWANK can be fixed soon, though, so we'll see.
*** There again, a check that a forward-package is not backward
would be very nice.
* Migrate from component-depends-on to action-depends-on
** I contend a future version of ASDF will replace
(component-depends-on operation component)
with (action-depends-on plan operation component).
This allows for different normalization strategies for dependencies
(including strategies that avoid resolving things to NIL),
a reified context for featurep checks, etc.
** Easy but long backward-compatible transition:
*** get all users to use the new gf and keep extending both new and old gf,
meanwhile the new gf has an around method that actually calls the old gf
except in testing mode and/or if we can check that they're using the new convention
*** when everyone has migrated, remove the old mode and the short-circuit.
** However, we cannot deprecate component-depends-on yet — not until we have
some transition in place to a better interface.
* Faster source-registry:
** In addition and/or as a substitute to the .cl-source-registry.cache,
that is meant to be semi-automatically managed, there could be
......@@ -457,10 +456,173 @@ It looks like SWANK can be fixed soon, though, so we'll see.
* Properly deprecate all that needs to go
** Get lisa, readable, bourbaki to NOT include ASDF 1 anymore in their sources.
** Identify all the things in ASDF that we want to deprecate, and make
sure we deprecate them all in 3.2.
sure we deprecate them all in 3.4.
** Implement support for deprecating old packages,
by having all functions and macros under the old package name be
proxies to those in the new package name?
(Exception: special variables can't be proxies, have to be imported.)
** Eradicate Quicklisp systems that depend on asdf-driver or
asdf-package-system, make them depend on uiop and asdf3.1 instead.
* Fix plan
** Goal: Implement proper incrementality for defsystem-depends-on.
Fix the bug wherein ASDF fails to properly handle incrementality for `defsystem-depends-on`.
https://bugs.launchpad.net/asdf/+bug/1500578
Some discussion about ASDF at
http://ccl.clozure.com/irc-logs/lisp/lisp-2016-10.txt
** DONE Items
*** Add `DEFINE-OP` for loading system definitions.
*** A `DEFINE-OP` will depend on other `DEFINE-OP`s
*** A `DEFINE-OP` depends on all `LOAD-OP` in the file.
Whether via defsystem-depends-on or manual `LOAD-OP`s.
*** Store timestamps in `DEFINE-OP` entries, not `*DEFINED-SYSTEMS*`.
*** In find-system, add condition that all dependencies are up-to-date.
*** This necessitates another layer of visit states,
that do not plan, just check planning...
*** `LOAD-ASD` calls `PERFORM DEFINE-OP` (not the other way around)
*** Systems probably need depend on `(DEFINE-OP . PRIMARY-SYSTEM)`
*** These dynamically-discovered dependencies are stored in a slot of `SYSTEM`.
*** Have a special kind of definition-depends-on dependency.
Before we reuse a node from a previous session,
we must make sure it is still properly defined.
This matters because we do reuse (at least *some* nodes, e.g. system nodes)
from previous sessions, and the control structure may have changed
(i.e. the system hasn't been redefined, but one of its defsystem-depends-on
dependencies has changed).
*** Handle incomplete definitions properly
Don't just let the incompletely parsed system be registered
in a false positive success as if it were an empty system.
Solution: It is registered, but considered out-of-date,
so next attempt to use it will cause a refresh attempt.
https://bugs.launchpad.net/asdf/+bug/1515372
*** Handle nested calls to `OPERATE`
We must accept that ASDF isn't really plan-then-perform,
as the ASDF1 model claimed to be.
It's an arbitrary nesting of plan-and-perform, like a parenthesization
with plan (steps) as left parenthesis and perform (steps) as right parenthesis,
and with operate being a maximal balanced span of plan and perform steps.
*** What goes in a shared dynamic `*SESSION*`, what in a private lexical `PLAN`?
Memoization CACHE of various functions, Status of VISITED-NODES, FORCING parameters.
The session will contain not just the current `*CACHE*`, but also
all or most of the graph traversal, thus the `:force` options, etc.
The `*PLAN*` may or may not contain some work queueing.
But even, e.g. a table of "nodes still directly blocking us"
is shared at the shared traversal level.
*** It is an error for a nested `OPERATE` to contradict session flags.
This raises the question of `:force` and `:force-not` flags.
An actual operate cannot contradict them and cause things to be performed
with a wrong subset of dependencies loaded.
Yet, `REQUIRED-COMPONENTS` (that does cause anything to be performed —
except via nested `OPERATE` from `defsystem-depends-on`)
must be able to skip some dependencies
(though maybe only _in addition_ to the global ones).
*** What timestamp for `forced` and `forced-not` actions?
If an action is forced-not, it should be returned with an action stamp from this cache,
and not with `nil` as is currently wrongly done
(unless of course the cache entry is missing,
at which point `nil` is the correct answer).
Conversely, a force means the stamp should be `t`
(indicating it needs to be done in the future)
(that part asdf gets right).
*** Handle planning things that are now done but were done in a previous phase of the build,
so dependency on them should trigger the build.
*** Protocol to not re-perform an action twice
In PERFORM-PLAN, check that an action wasn't already DONE-P due to
being needed in a previous phase of recursively calling OPERATE,
before to PERFORM it.
To correctly propagate timestamps in across nested calls,
the graph traversal and timestamp propagation is shared at the session level
between all nested calls to `OPERATE`.
During a session and across phases, every action has a status made of
a stamp and three bits: KEEP-P, DONE-P, NEED-P.
The stamp also survives across sessions, in `COMPONENT-OPERATION-TIMES`.
*** Detect circularities across nested calls to `OPERATE`
This means the circularity detection data is part of the session.
Notably, the visiting list and set.
The top of the list can also serve to record the dependencies from `OPERATE` to `DEPEND-OP`.
*** In UIOP, RECORD-DEPENDENCY must not cross to an upper plan and/or it should
record the dependency in said upper plan (that matters because
parallel plans doing extra checking, and this is a problem
whereby some links to extra actions are left unresolved).
Maybe in the future, record the dependency in the session, not in a plan.
** TODO items
*** find-system should probably cause a dependency on define-op, especially inside another define-op
*** Is it worth it trying to optimize away a dependency on a define-op
when an action already depends on another operation that "obviously" depends on that define-op?
*** Document DEFINE-OP and the design in asdf.texinfo and/or in some article.
** Future developments
*** Grovel all .asd files in Quicklisp and collect all violations of declarativeness,
where declarativeness is having only defsystem forms. Specially classify in-package,
maybe also defpackage. Then we can assess the impact of requiring declarativeness.
*** Also grovel Quicklisp for all extensions to ASDF:
see if there are new methods on perform, input-files, output-files, operate, etc.
*** To allow defsystem-depends-on to another secondary system within the
same file, I can imagine some system to dynamically create a separate action
(DEFINE-OP foo/x) for each form defsystem foo/x, and have it depend on
all the preceding actions operated in the .asd file.
Plus a last action for dangling load-system or operate statements?
Or one action for every operate statement?
There would be no action of the entire load-asd, but a separate mechanism
to detect circularity?
What about dangling statements? They would be a bit like xcvb weakly-ordered
run-time dependencies that CAN have circularities? Ouch. Really ugly.
Each statement would need its own action? Ouch.
*** Have load-asd clear any invalidated previous secondary system
before loading, which supposes remembering what these systems are.
Then, you're not allowed to forward-reference a secondary system that
hasn't been defined yet while you're loading the .asd, whereas you
can load the asd if you aren't loading it yet.
*** Maybe split DEFINE-OP into PREPARE-DEFINE-OP and DEFINE-OP, so that
secondary systems can be properly loaded?
*** Add mapping from files to action that creates it and/or uses it?
Then we can detect conflict between multiple actions creating one file.
Forbid direct use of files not listed as inputs? What about transitive inputs?
Build in a container that has only the proper files...
*** What data structure for scheduling planned actions?
A doubly-linked to remove entire chunks of the plan after a sub-operate?
That won't do, because later-discovered defsystem dependency can refer to
systems that were planned but not performed, so there need be no contiguity
in the chunks that are performed by subcalls to operate.
Actually, if we otherwise forbid reference to another system
from non-system components, then we could move things with system granularity,
which is one thing; but we still need a general mechanism at the system level.
So, whether it's an explicit queue or an implicit traversal of the dependency graph,
we must check whether an action has already been done before we try to perform it.
And an implicit traversal has the advantage that you can avoid entire subtrees
when you find that the current node has already been performed during
a previous phase of the plan.
In conclusion: "just" traverse the graph at the last minute, and
perform things serially, and/or if you want some parallelism,
build a queue dynamically at the last minute for the "now" jobs
while keeping most of the plan as graph structures.
*** Merging in parts of POIU?
ASDF internals are nothing to be proud of:
ASDF1 was optimized for "smallest code that kind of works in the usual case",
and ASDF3 was optimized for "smallest code that fixes ASDF1 in the general case
while mostly maintaining backward compatibility".
Because of its optimization constraint, ASDF1 was using and abusing lists a lot,
rather than defining nice data structures.
ASDF3 uses somewhat more structured data, with a few more classes, and
algorithms that are O(n) instead of O(n**3) in simple cases and
exponential in carefully crafted examples;
yet it has shunned any definition of more advanced or general data structures.
By contrast, POIU has a dequeue to represent queued compilation jobs, and
a general graph representation for the action graph
(though the original POIU was also O(n**3) because it lacked PREPARE-OP).
I could import the POIU queue and graph representations into ASDF, and
that would add about 100 lines of code to UIOP for the queue, and
200 lines to ASDF for the graph, maybe a bit more if the code is
generalized, commented and largely put in UIOP.
Then, there's the support for actual forking and using forks,
that would be about 400 lines added to UIOP, and
the using it in ASDF to consume the plan, which would be under 200 lines.
But those parts are probably better left out of ASDF itself.
That said, it all was a lot when ASDF was < 1000 lines of code,
but isn't all that much now that it is > 12000 lines of code.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment