#+LaTeX_CLASS: beamer #+LaTeX_CLASS_OPTIONS: [presentation] #+BEAMER_FRAME_LEVEL: 2 #+TITLE: Evolving ASDF #+DATE: International Lisp Conference, 19 October 2010 #+AUTHOR: François-René Rideau \and Robert P. Goldman #+BEAMER_HEADER_EXTRA: \usetheme{default}\usecolortheme{default} \subtitle{More cooperation, less coordination} \author{\begin{tabular} \ François-René Rideau \\ ITA Software \\ Cambridge, MA \end{tabular} \and \begin{tabular} \ Robert P. Goldman \\ SIFT, LLC \\ Minneapolis, MN \end{tabular}} \AtBeginSection[]{\begin{frame} \frametitle{Outline} \tableofcontents[currentsection]\end{frame}} #+COLUMNS: %45ITEM %10BEAMER_env(Env) %10BEAMER_envargs(Env Args) %4BEAMER_col(Col) %8BEAMER_extra(Extra) #+PROPERTY: BEAMER_col_ALL 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 :ETC * COMMENT To do items ** DONE Center institutions under authors ** DONE Finish "Future Directions" [[*Future directions]] ** TODO Fix "Pathname limitations" slide *** Needs structure *** Table? Something to make the before and after stand out *** Add the missing trailing slash in module pathnames to the slide... ** TODO Missing extension hooks [[*Missing Extension Hooks]] *** Can we say what we did to fix this? Extend the published API? *** Possible bonus: Anything broadly interesting about the change? + Take home message about how to build your next API? + Ease/difficulty of changing API? OK if we don't have anything... * COMMENT Possible theme for enhancements section A number of things (returning values, handling cross-system dependencies properly) would be simplified if the ASDF plans were hierarchically (tree) structured, instead of being just flat sequences. Plans would have a flat, sequential form for execution, but there would be a superstructure that could accumulate values, etc. This would also help give operations on modules and systems a more intuitive semantics. Could also help let people write useful =:around= methods. Question: could we weave this into the concluding part of the talk. *POSSIBLY* --- propose not to do this unless it can be done reasonably clearly, and there may not be enough time remaining. * COMMENT Org-principles for this document ** Just like outline mode --- More "*" in header means deeper nesting ** Simple markup --- if you like! --- we can add this later *** /slashes/ for ital *** *stars* for bold *** =equal= signs for monospace ** Slide structure *** Single-star headings are sections *** Double-star headings are individual slides For this reason there are some things that look odd, where we have only a single sub-bullet under a major header. ** Outline proper starts in the following items * COMMENT Structure ** Suggest we have 15-18 slides ** I have 20 slides in the current draft * Summary ** COMMENT Two slides here, for context ** ASDF *** *build system*: compile source files **** a bit like =make= for C or =ant= for Java *** *in image*: also load software **** totally unlike either =make= or =ant= **** maintain long-lived system state *** *declarative*: describe system dependencies **** not imperative instructions on how to build *** *specialized*: oriented toward Lisp software **** not for arbitrary tasks with dependencies *** *extensible*: can learn new classes of files **** not for arbitrary tasks with dependencies *** a key piece of CL *community* plumbing ** /Evolving/ ASDF *** Fixing up ASDF **** numerous issues: pathname limitations, difficult configuration, dependency bugs, fasl mess, portability, missing extension hooks *** ... not wrecking it in the process **** numerous existing users: some depend on obscure features, existing bugs, undocumented internals... *** How is evolving ASDF possible? *** We discovered interesting challenges **** technical: hot upgrade **** social: responsibility over upgrade, configuration *** Development principles **** some very well known, some less well known * What is ASDF? ** COMMENT 4 slides here ** Dominant build system for CL ***** COMMENT what is this thing below about? ***** COMMENT Not always going back to a clean slate like other tools *** Built on the shoulders of giants **** =MK-DEFSYSTEM= **** Symbolics and other proprietary =DEFSYSTEM= versions **** kmp's MIT AI Memo 801, rer's MIT AI TR 874. *** Brilliant key idea establishes ASDF dominance **** Use =*load-truename*= to find system component files **** Suddenly, installing CL systems is easy + No more wrestling with logical pathnames + Especially since logical pathnames are insufficiently portable ** ASDF system definitions *** COLUMNS :B_columns: :PROPERTIES: :BEAMER_env: columns :END: **** 0.4 :B_column: :PROPERTIES: :BEAMER_env: column :END: + Contain /metadata/ + Contain /components/ + Contain /dependencies/ - external - internal **** 0.6 :B_column: :PROPERTIES: :BEAMER_env: column :END: #+BEAMER: \small #+BEGIN_SRC common-lisp (defsystem "foo" :version "1.0.1" :author "Fare and RPG" :depends-on (cl-ppcre) :components ((:file "blah") (:module "mod" :components ((:file "bar") (:file"baz") (:file "quux"))))) #+END_SRC ** How does ASDF work? #+BEGIN_BEAMER \only<1>{\centerline{\includegraphics[width=10cm]{talk-images/find-system}}} \only<2>{\centerline{\includegraphics[width=10cm]{talk-images/defsystem}}} \only<3>{\centerline{\includegraphics[width=10cm]{talk-images/traverse}}} \only<4>{\centerline{\includegraphics[width=10cm]{talk-images/execute}}} #+END_BEAMER ** Three things I wish you knew about ASDF 1. ASDF doesn't just build: it makes a /plan/ to build, and then executes the plan 2. Operating on a module is not /wrapped around/ the operations on components + =:around= methods don't work for customization 3. You can tell ASDF about system versions + Say what the version of your system is + $x.y.z$, please + Please don't skimp on the high-order numbers or ASDF won't detect incompatibilities + Say what library versions it depends on * Fixing ASDF ** What was wrong with ASDF *** pathname limitations *** difficult configuration *** dependency bugs *** fasl mess *** portability *** missing extension hooks * Pathname limitations ** Pathname Limitations *** *Core idea:* #p"foo/bar" can never be portable. "foo/bar" can be. *** (:file "foo") ==> #p"foo.lisp" *** (:file "foo.bar") ==> before: #p"foo.bar" ASDF2: #p"foo.bar.lisp" *** (:file "foo-V1.2") ==> before: #p"foo-V1.2" ASDF2: #p"foo-V1.2.lisp" *** (:file "foo/bar") ==> before: non portable ASDF2: #p"foo/bar.lisp" **** (:file "foo-bar" :pathname #.(merge-pathnames (make-pathname :name "bar" :directory '(:relative "foo") :type "lisp" :defaults \*load-truename\*) \*load-truename\*)) *** (:static-file "README") ==> #p"README" *** (:file "README" :pathname #p"README") ==> before: #p"README.lisp" ASDF2: #p"README" ** Pathname fixes *** CL pathname FAIL: underspecified semantics means not portably usable **** Solution: provide a portable *abstraction* ***** parse strings into "relative pathnames with optionally specified type" *** ASDF FAIL: it was hard to get it actually right **** Solution: make it hard for the user to get it wrong *** ASDF FAIL: it was hard to know you were wrong **** Solution: better fail early for everyone than pass as working for some *** ASDF2 is much better, but still trips users **** Solution: keep improving code, error cases, documentation. ** Configuration Difficulties *** ** Output locations ** Dependency Bugs *** Changes within a module didn't cause recompilation of dependent components *** Changes on module dependencies didn't cause recompilation of module components *** Remember: systems are modules! *** Partially fixed --- see future directions ** Portability ** Missing Extension Hooks *** POIU (Parallel extension) needed to redefine some classes and methods *** ITA needed some extensions for printing compilation progress. * Hot-patching ASDF ** COMMENT 4 slides here ** Hot-patching ASDF *** Loading ASDF into a running CL image containing ASDF *** Why is this critical? **** If people can't load a new ASDF on top of an old one, they can never write portable code using new features **** /Unless/ all the implementations get together and /simultaneously/ update their packaged versions **** Hot-patching allows ASDF to evolve *** Why is this hard? **** We are replacing bits of ASDF /while it is running/ **** While it is running /to build and load itself/ **** COMMENT image of snake eating its own tail... ** Hot-patching: how to do it *** Mostly easy: upgrading data, with =update-instance-for-redefined-class= **** "pull" model: everything sees the new schema. No "push" stages. *** Hard: changing a function's signature **** CLOS won't even let you do it. *** Rebinding (=fmakunbound=) vs Shadowing (=unintern= in all packages) **** No "rebind to transitional version that wraps the new version, then shadow" *** Harder: modified functions are in the continuation of =compile-file= or =load= **** Can't update continuation frames; they have no first-class names to be rebound. **** Worse if multithreaded: no atomicity * Configuration ** COMMENT 4 slides here ** Configuring ASDF *** Need to be able to find systems **** We can find system components from the system definition **** But we still need to find the system definitions! *** Need to place compiled files /Explain the need here/ * Future directions ** COMMENT 2 slides here ** Future directions *** There is still a lot of room for improvement with ASDF *** Bugs *** New features *** Better documentation ** Outstanding bug: across system dependencies do not properly trigger recompilations + We /did/ fix a related bug --- module dependencies didn't work in ASDF 1 + Challenging to fix + Some think it's not a bug + Systems/libraries should provide stable APIs + Across sessions, ASDF doesn't have the information it needs + Not just FASLs newer than source files + What was the state of the image /when system FOO was compiled/? What was loaded? + Systems and modules are not components like others ** Possible new features *** More declarative =DEFSYSTEM= **** Current procedural aspects make introspection difficult **** Need procedural code to invoke ASDF extensions **** Would require modifying how DEFSYSTEM is read **** What about side-effect isolation? ***** See XCVB *** Make =TRAVERSE= part of the API **** Pro: Would allow cleaner extensions **** Con: Might lock in buggy behavior *** Conditional components **** E.g., only load this file if you are doing the =TEST-OP= *** More operations (e.g., =DOC-OP=) *** Return values **** =OPERATE= doesn't return a value --- bad for, e.g., testing **** Complicated by linear plans --- no hierarchical structure for collecting return values *** COMMENT --- possible theme here --- having /hierarchical/ plans might make a lot of things easier. ** What is still wrong with ASDF *** Configuration is still harder than ideal **** We tried to accommodate every previous extension variant. **** Instead, should we have authoritatively standardized good defaults? **** Or does our documentation just stink? *** FASLs still not separate enough **** bit us with ASDF 2.006 *** The documentation *** Hard to extend **** At least partially due to the (lack of) documentation *** ASDF2 is still not universally available * Conclusions ** COMMENT 1 slide here ** ASDF 2 is now available in stores near you... *** =http://common-lisp.net/project/asdf/= *** Download and load into your implementation of choice *** Bundled with an increasing number of implementations **** COLUMNS :B_columns: :PROPERTIES: :BEAMER_env: columns :END: ***** 0.5 :B_column: :PROPERTIES: :BEAMER_env: column :END: ****** SBCL ****** ABCL ****** ECL ****** SCL ****** CMUCL ****** CCL ***** 0.5 :B_column: :PROPERTIES: :BEAMER_env: column :END: ****** Yet to come (?) ******* CLISP (discussed on the list) ******* ACL ******* LispWorks ******* Corman (no response) ***** COMMENT end of columns *** =:asdf2= in =*features*= ** Conclusions *** ASDF 2 needs new maintainers *** But please remember --- ASDF is the glue that holds the CL community together **** Be gentle with it! ** Thanks *** ASDF: Daniel Barlow, Christophe Rhodes, Gary King, ... **** A dozen contributors *** ASDF2: James Anderson, Juan-Jose Garcia-Ripoll *** Paper: Vadim Nasardinov, Ethan Schwartz, Scott McKay, Dan Barlow, anonymous reviewers *** All our users for testing --- and for building Lisp software! **** notable testing by Zach Beane, Samium Gromoff, Daniel Herring... ** Unupgradability of ASDF *** To upgrade ASDF... **** ASDF1: deep configuration before startup, in every implementation, every program **** ASDF2: just install ASDF, no configuration unless non-standard location **** you may rely on a feature... ***** ASDF1: only if all supported implementations have upgraded ***** ASDF2: anytime, ship recent ASDF if a feature isn't universally installed **** it brings value for an implementation to upgrade... ***** ASDF1: only after everybody else agreed to upgrade ***** ASDF2: anytime, or the implementation will lag behind **** After a given implementation upgrades... ***** ASDF1: more disuniformity in features available in installed base ***** ASDF2: more uniformity in features available in installed base ** First, fix upgrade incentives *** Dynamics of coordination **** In a stable process, each iteration provides more coordination than it requires *** Allowing for divergence creates an incentive towards convergence ** General Lessons Learned (Summary) **** ASDF is an ``in-image'' build system managing systems that are compiled and loaded in the current {\CL} image **** CL makes dynamic data upgrade extraordinarily easy. **** CL support for hot upgrade of code may exist but is anything but seamless **** the general problem with {\CL} is that its semantics are defined in terms of irreversible side-effects to global data structures in the current image **** Anything that can be provided as an extension should be provided as an extension and left out of the core **** We can't simply be ``transparent'' with respect to semantic discrepancies between underlying implementations; we must abstract those discrepancies away. **** have the person who knows write the configuration, not the person who doesn't **** good data structures and algorithms matter **** thou shalt tailor thy datastructures to the target problem **** Without the test suite, we'd be nowhere.