From 7f4c8134bf354628210930e6c89f25a2fb5e381a Mon Sep 17 00:00:00 2001
From: Francois-Rene Rideau <fare@tunes.org>
Date: Sat, 22 Oct 2016 17:57:26 -0400
Subject: [PATCH] Update TODO wrt support for proper phase separation

---
 TODO | 258 ++++++++++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 210 insertions(+), 48 deletions(-)

diff --git a/TODO b/TODO
index 1899bc60b..bb0303d6d 100644
--- a/TODO
+++ b/TODO
@@ -1,6 +1,6 @@
 * Decree that versions older than 2 years are not supported;
-   yet, keep supporting an upgrade from whatever vendors provide
-   — so push them to provide recent stuff.
+  yet, keep supporting an upgrade from whatever vendors provide
+  — so push them to provide recent stuff.
 ** Already, swank-asdf doesn't support anything older than 2.014.6
    (as provided by the first 2011 iteration of Quicklisp), and
    I (fare) am pushing for swank-asdf to not support anything older
@@ -17,7 +17,13 @@
 ** Make tests stateless and/or make any modifications private,
    so that they can be run in parallel.
 
-* Moving toward cross-compilation (ASDF 3.2? 4?)
+* Testing support
+** Replace test-op with run-test-op and test-report-op ?
+
+* Implement a style-checker, that will issue STYLE-WARNINGs
+  when you don't follow all the recommended guidelines.
+
+* Moving toward cross-compilation (ASDF 3.4? 4?)
    This would allow to get most of the benefits of XCVB
    (reproducibility, scalability, applicability to mobile platforms, etc.),
    except without the backward incompatibility of XCVB.
@@ -112,14 +118,14 @@
     make t l=cmucl t=test-encodings.script
     which was a different PCL bug from the above (without same workaround)
     but possibly also triggered by the non-standard declaim.
-*** `#5(1 ,@`(2 3)))` returns #(1 2 3),
-     rather than #(1 2 3 2 3 2 3 2 3) or even better #(1 2 3 3 3).
+*** `#5(1 ,@\`(2 3)))` returns `#(1 2 3)`,
+     rather than `#(1 2 3 2 3 2 3 2 3)` or even better `#(1 2 3 3 3)`.
 *** It purports to support Unicode, but it has only 16-bit characters.
    Note that (subtypep 'character 'base-char) says T -- everything is a base char(!)
 
 ** SCL has bugs:
 *** it doesn't like run-program from a modified directory?
-*** it somehow pushes :non-base-chars-exist-p even though +non-base-chars-exist-p+ is NIL???
+*** it somehow pushes `:non-base-chars-exist-p` even though `+non-base-chars-exist-p+` is `NIL`???
 
 ** XCL has bad bugs:
 *** make-pathname doesn't handle :type nil properly and
@@ -140,6 +146,7 @@
    (make-pathname :name "foo" :type :unspecific) ==> #P"foo."
     If bug is ever fixed, upgrade *unspecific-pathname-type* in uiop/pathname.
 *** XCL doesn't properly process files the name of which contains a *
+*** XCL fails to recognize that specialized method parameters are implicitly ignorable.
 
 ** GCL is almost working again; but implementation bugs remain.
    See Francois-Rene Rideau's messages on gcl-devel starting November 2013.
@@ -154,33 +161,33 @@
    INTERNAL-SIMPLE-ERROR: The package named ASDF/INTERFACE, does not exist
 *** Another GCL compiler bug:
    when I changed the definition of getcwd from
-   (let ((*default-pathname-defaults* #p"")) (truename #p"")) to
-   (let ((*default-pathname-defaults* *nil-pathname*)) (truename *nil-pathname*))
+   `(let ((*default-pathname-defaults* #p"")) (truename #p""))` to
+   `(let ((*default-pathname-defaults* *nil-pathname*)) (truename *nil-pathname*))`
    to guard against e.g. a logical-pathname context while loading asdf
-   and parsing #p"", calls to getcwd result in a segfault.
+   and parsing `#p""`, calls to `getcwd` result in a segfault.
 *** An another bug: gcl refuses dynamic-extent declaration on functions.
-   uiop/stream.lisp:         #-gcl (declare (dynamic-extent ,@(when
-   before `(#',beforef)) ,@(when after `(#',afterf))))
-*** (typep p 'logical-pathname) should be T if p has a logical-pathname host.
-*** apropos is case-sensitive and returns a same symbol many times
+   ```uiop/stream.lisp:         #-gcl (declare (dynamic-extent ,@(when
+   before `(#',beforef)) ,@(when after `(#',afterf))))```
+*** `(typep p 'logical-pathname)` should be T if p has a logical-pathname host.
+*** `apropos` is case-sensitive and returns a same symbol many times
    (presumably once per package where it is present,
    instead of just once for its home package)
-*** compile-file fails to return proper secondary values in case of non-style WARNING.
-*** (pathname-directory #p"foo/") is incorrectly ("foo") instead of (:RELATIVE "foo")
+*** `compile-file` fails to return proper secondary values in case of non-style `WARNING`.
+*** `(pathname-directory #p"foo/")` is incorrectly `("foo")` instead of `(:RELATIVE "foo")`
 *** Do whatever it takes to pass the asdf tests, add the above?
-*** Trying to uiop:slurp-stream-forms from a stream with #+(or) :foo
-   (or read-file-forms from an file with same) results in an error,
-   rather than nil. This is probably a bug in #+ processing.
+*** Trying to `uiop:slurp-stream-forms` from a stream with `#+(or) :foo`
+   (or `read-file-forms` from an file with same) results in an error,
+   rather than `nil`. This is probably a bug in `#+` processing.
    Unhappily, debian creates such a file in
-   /etc/common-lisp/asdf-output-translations.conf.d/01-common-lisp-controller.conf
+   `/etc/common-lisp/asdf-output-translations.conf.d/01-common-lisp-controller.conf`
 *** Tests that try to catch an error fail (but catching a warning succeeds),
    which suggests brokenness in handler-bind and/or error.
-*** COMPILE-FILE* fails due to handler-bind error in
-   with-muffled-compiler-conditions or so.
-*** `#5(1 ,@`(2 3)))` returns #(1 2 3),
-     rather than #(1 2 3 2 3 2 3 2 3) or even better #(1 2 3 3 3).
-*** (DIRECTORY #p"*.*") fails to match files with pathname type NIL.
-*** GCL doesn't properly process files the name of which contains a *
+*** `COMPILE-FILE*` fails due to handler-bind error in
+   `with-muffled-compiler-conditions` or so.
+*** `#5(1 ,@`(2 3)))` returns `#(1 2 3)`,
+     rather than `#(1 2 3 2 3 2 3 2 3)` or even better `#(1 2 3 3 3)`.
+*** `(DIRECTORY #p"*.*")` fails to match files with pathname-type `NIL`.
+*** GCL doesn't properly process files the name of which contains a `*`
 
 ** ABCL has a few bugs.
 *** ABCL fails the timestamp propagation test.
@@ -243,12 +250,6 @@
 ** Real solution: defer parsing and evaluation of defsystem forms?
    Or simply detect circularity and issue an appropriate error?
 
-* Implement proper incrementality for defsystem-depends-on.
-** For proper incremental redefinitions, systems probably need to
-   depend-on (load-asd-op . primary-system) which itself will
-   depend-on all the load-op of defsystem-depends-on in that file.
-   https://bugs.launchpad.net/asdf/+bug/1500578
-
 * Include some ABL test for stassats's (now obsolete?) thing:
    (asdf:enable-asdf-binary-locations-compatibility
    :centralize-lisp-binaries t :default-toplevel-directory *fasl-dir*)
@@ -382,6 +383,20 @@ It looks like SWANK can be fixed soon, though, so we'll see.
    version
 
 * ASDF4: search for this tag, rename things (incompatibly, thus) and cleanup code.
+** Migrate from component-depends-on to action-depends-on
+*** I contend a future version of ASDF will replace
+`(component-depends-on operation component)`
+with `(action-depends-on plan operation component)`.
+This allows for different normalization strategies for dependencies
+(including strategies that avoid resolving things to NIL),
+a reified context for featurep checks, etc.
+*** Easy but long backward-compatible transition:
+**** get all users to use the new gf and keep extending both new and old gf,
+   meanwhile the new gf has an around method that actually calls the old gf
+   except in testing mode and/or if we can check that they're using the new convention
+**** when everyone has migrated, remove the old mode and the short-circuit.
+*** However, we cannot deprecate component-depends-on yet — not until we have
+   some transition in place to a better interface.
 
 * Documentation
 ** See message from 2014-01-27 05:26:44 GMT for stuff to document.
@@ -417,22 +432,6 @@ It looks like SWANK can be fixed soon, though, so we'll see.
 *** There again, a check that a forward-package is not backward
    would be very nice.
 
-* Migrate from component-depends-on to action-depends-on
-** I contend a future version of ASDF will replace
-   (component-depends-on operation component)
-   with (action-depends-on plan operation component).
-   This allows for different normalization strategies for dependencies
-   (including strategies that avoid resolving things to NIL),
-   a reified context for featurep checks, etc.
-** Easy but long backward-compatible transition:
-*** get all users to use the new gf and keep extending both new and old gf,
-   meanwhile the new gf has an around method that actually calls the old gf
-   except in testing mode and/or if we can check that they're using the new convention
-*** when everyone has migrated, remove the old mode and the short-circuit.
-** However, we cannot deprecate component-depends-on yet — not until we have
-   some transition in place to a better interface.
-
-
 * Faster source-registry:
 ** In addition and/or as a substitute to the .cl-source-registry.cache,
    that is meant to be semi-automatically managed, there could be
@@ -457,10 +456,173 @@ It looks like SWANK can be fixed soon, though, so we'll see.
 * Properly deprecate all that needs to go
 ** Get lisa, readable, bourbaki to NOT include ASDF 1 anymore in their sources.
 ** Identify all the things in ASDF that we want to deprecate, and make
-   sure we deprecate them all in 3.2.
+   sure we deprecate them all in 3.4.
 ** Implement support for deprecating old packages,
    by having all functions and macros under the old package name be
    proxies to those in the new package name?
    (Exception: special variables can't be proxies, have to be imported.)
 ** Eradicate Quicklisp systems that depend on asdf-driver or
    asdf-package-system, make them depend on uiop and asdf3.1 instead.
+
+* Fix plan
+** Goal: Implement proper incrementality for defsystem-depends-on.
+Fix the bug wherein ASDF fails to properly handle incrementality for `defsystem-depends-on`.
+https://bugs.launchpad.net/asdf/+bug/1500578
+Some discussion about ASDF at
+http://ccl.clozure.com/irc-logs/lisp/lisp-2016-10.txt
+
+** DONE Items
+*** Add `DEFINE-OP` for loading system definitions.
+*** A `DEFINE-OP` will depend on other `DEFINE-OP`s
+*** A `DEFINE-OP` depends on all `LOAD-OP` in the file.
+Whether via defsystem-depends-on or manual `LOAD-OP`s.
+*** Store timestamps in `DEFINE-OP` entries, not `*DEFINED-SYSTEMS*`.
+*** In find-system, add condition that all dependencies are up-to-date.
+*** This necessitates another layer of visit states,
+that do not plan, just check planning...
+*** `LOAD-ASD` calls `PERFORM DEFINE-OP` (not the other way around)
+*** Systems probably need depend on `(DEFINE-OP . PRIMARY-SYSTEM)`
+*** These dynamically-discovered dependencies are stored in a slot of `SYSTEM`.
+*** Have a special kind of definition-depends-on dependency.
+Before we reuse a node from a previous session,
+we must make sure it is still properly defined.
+This matters because we do reuse (at least *some* nodes, e.g. system nodes)
+from previous sessions, and the control structure may have changed
+(i.e. the system hasn't been redefined, but one of its defsystem-depends-on
+dependencies has changed).
+*** Handle incomplete definitions properly
+Don't just let the incompletely parsed system be registered
+in a false positive success as if it were an empty system.
+Solution: It is registered, but considered out-of-date,
+so next attempt to use it will cause a refresh attempt.
+https://bugs.launchpad.net/asdf/+bug/1515372
+*** Handle nested calls to `OPERATE`
+We must accept that ASDF isn't really plan-then-perform,
+as the ASDF1 model claimed to be.
+It's an arbitrary nesting of plan-and-perform, like a parenthesization
+with plan (steps) as left parenthesis and perform (steps) as right parenthesis,
+and with operate being a maximal balanced span of plan and perform steps.
+*** What goes in a shared dynamic `*SESSION*`, what in a private lexical `PLAN`?
+Memoization CACHE of various functions, Status of VISITED-NODES, FORCING parameters.
+The session will contain not just the current `*CACHE*`, but also
+all or most of the graph traversal, thus the `:force` options, etc.
+The `*PLAN*` may or may not contain some work queueing.
+But even, e.g. a table of "nodes still directly blocking us"
+is shared at the shared traversal level.
+*** It is an error for a nested `OPERATE` to contradict session flags.
+This raises the question of `:force` and `:force-not` flags.
+An actual operate cannot contradict them and cause things to be performed
+with a wrong subset of dependencies loaded.
+Yet, `REQUIRED-COMPONENTS` (that does cause anything to be performed —
+except via nested `OPERATE` from `defsystem-depends-on`)
+must be able to skip some dependencies
+(though maybe only _in addition_ to the global ones).
+*** What timestamp for `forced` and `forced-not` actions?
+If an action is forced-not, it should be returned with an action stamp from this cache,
+and not with `nil` as is currently wrongly done
+(unless of course the cache entry is missing,
+at which point `nil` is the correct answer).
+Conversely, a force means the stamp should be `t`
+(indicating it needs to be done in the future)
+(that part asdf gets right).
+*** Handle planning things that are now done but were done in a previous phase of the build,
+so dependency on them should trigger the build.
+*** Protocol to not re-perform an action twice
+In PERFORM-PLAN, check that an action wasn't already DONE-P due to
+being needed in a previous phase of recursively calling OPERATE,
+before to PERFORM it.
+To correctly propagate timestamps in across nested calls,
+the graph traversal and timestamp propagation is shared at the session level
+between all nested calls to `OPERATE`.
+During a session and across phases, every action has a status made of
+a stamp and three bits: KEEP-P, DONE-P, NEED-P.
+The stamp also survives across sessions, in `COMPONENT-OPERATION-TIMES`.
+*** Detect circularities across nested calls to `OPERATE`
+This means the circularity detection data is part of the session.
+Notably, the visiting list and set.
+The top of the list can also serve to record the dependencies from `OPERATE` to `DEPEND-OP`.
+*** In UIOP, RECORD-DEPENDENCY must not cross to an upper plan and/or it should
+record the dependency in said upper plan (that matters because
+parallel plans doing extra checking, and this is a problem
+whereby some links to extra actions are left unresolved).
+Maybe in the future, record the dependency in the session, not in a plan.
+
+** TODO items
+*** find-system should probably cause a dependency on define-op, especially inside another define-op
+*** Is it worth it trying to optimize away a dependency on a define-op
+when an action already depends on another operation that "obviously" depends on that define-op?
+*** Document DEFINE-OP and the design in asdf.texinfo and/or in some article.
+
+** Future developments
+*** Grovel all .asd files in Quicklisp and collect all violations of declarativeness,
+where declarativeness is having only defsystem forms. Specially classify in-package,
+maybe also defpackage. Then we can assess the impact of requiring declarativeness.
+*** Also grovel Quicklisp for all extensions to ASDF:
+see if there are new methods on perform, input-files, output-files, operate, etc.
+*** To allow defsystem-depends-on to another secondary system within the
+same file, I can imagine some system to dynamically create a separate action
+(DEFINE-OP foo/x) for each form defsystem foo/x, and have it depend on
+all the preceding actions operated in the .asd file.
+Plus a last action for dangling load-system or operate statements?
+Or one action for every operate statement?
+There would be no action of the entire load-asd, but a separate mechanism
+to detect circularity?
+What about dangling statements? They would be a bit like xcvb weakly-ordered
+run-time dependencies that CAN have circularities? Ouch. Really ugly.
+Each statement would need its own action? Ouch.
+*** Have load-asd clear any invalidated previous secondary system
+before loading, which supposes remembering what these systems are.
+Then, you're not allowed to forward-reference a secondary system that
+hasn't been defined yet while you're loading the .asd, whereas you
+can load the asd if you aren't loading it yet.
+*** Maybe split DEFINE-OP into PREPARE-DEFINE-OP and DEFINE-OP, so that
+secondary systems can be properly loaded?
+*** Add mapping from files to action that creates it and/or uses it?
+Then we can detect conflict between multiple actions creating one file.
+Forbid direct use of files not listed as inputs? What about transitive inputs?
+Build in a container that has only the proper files...
+*** What data structure for scheduling planned actions?
+A doubly-linked to remove entire chunks of the plan after a sub-operate?
+That won't do, because later-discovered defsystem dependency can refer to
+systems that were planned but not performed, so there need be no contiguity
+in the chunks that are performed by subcalls to operate.
+Actually, if we otherwise forbid reference to another system
+from non-system components, then we could move things with system granularity,
+which is one thing; but we still need a general mechanism at the system level.
+So, whether it's an explicit queue or an implicit traversal of the dependency graph,
+we must check whether an action has already been done before we try to perform it.
+And an implicit traversal has the advantage that you can avoid entire subtrees
+when you find that the current node has already been performed during
+a previous phase of the plan.
+In conclusion: "just" traverse the graph at the last minute, and
+perform things serially, and/or if you want some parallelism,
+build a queue dynamically at the last minute for the "now" jobs
+while keeping most of the plan as graph structures.
+
+*** Merging in parts of POIU?
+ASDF internals are nothing to be proud of:
+ASDF1 was optimized for "smallest code that kind of works in the usual case",
+and ASDF3 was optimized for "smallest code that fixes ASDF1 in the general case
+while mostly maintaining backward compatibility".
+
+Because of its optimization constraint, ASDF1 was using and abusing lists a lot,
+rather than defining nice data structures.
+ASDF3 uses somewhat more structured data, with a few more classes, and
+algorithms that are O(n) instead of O(n**3) in simple cases and
+exponential in carefully crafted examples;
+yet it has shunned any definition of more advanced or general data structures.
+By contrast, POIU has a dequeue to represent queued compilation jobs, and
+a general graph representation for the action graph
+(though the original POIU was also O(n**3) because it lacked PREPARE-OP).
+
+I could import the POIU queue and graph representations into ASDF, and
+that would add about 100 lines of code to UIOP for the queue, and
+200 lines to ASDF for the graph, maybe a bit more if the code is
+generalized, commented and largely put in UIOP.
+
+Then, there's the support for actual forking and using forks,
+that would be about 400 lines added to UIOP, and
+the using it in ASDF to consume the plan, which would be under 200 lines.
+But those parts are probably better left out of ASDF itself.
+That said, it all was a lot when ASDF was < 1000 lines of code,
+but isn't all that much now that it is > 12000 lines of code.
-- 
GitLab