talk-outline.org 14.9 KB
 Robert P. Goldman committed Sep 30, 2010 1 2 3 #+LaTeX_CLASS: beamer #+LaTeX_CLASS_OPTIONS: [presentation] #+BEAMER_FRAME_LEVEL: 2  Robert P. Goldman committed Oct 14, 2010 4 5 #+TITLE: Evolving ASDF #+DATE: International Lisp Conference, 19 October 2010  Francois-Rene Rideau committed Oct 18, 2010 6 7 #+AUTHOR: François-René Rideau \and Robert P. Goldman #+BEAMER_HEADER_EXTRA: \usetheme{default}\usecolortheme{default} \subtitle{More cooperation, less coordination} \author{\begin{tabular} \ François-René Rideau \\ ITA Software \\ Cambridge, MA \end{tabular} \and \begin{tabular} \ Robert P. Goldman \\ SIFT, LLC \\ Minneapolis, MN \end{tabular}} \AtBeginSection[]{\begin{frame} \frametitle{Outline} \tableofcontents[currentsection]\end{frame}}  Robert P. Goldman committed Sep 30, 2010 8 9 10 #+COLUMNS: %45ITEM %10BEAMER_env(Env) %10BEAMER_envargs(Env Args) %4BEAMER_col(Col) %8BEAMER_extra(Extra) #+PROPERTY: BEAMER_col_ALL 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 :ETC  Robert P. Goldman committed Oct 14, 2010 11 12 * COMMENT To do items  Francois-Rene Rideau committed Oct 17, 2010 13 ** DONE Center institutions under authors  Robert P. Goldman committed Oct 17, 2010 14 ** DONE Finish "Future Directions"  Francois-Rene Rideau committed Oct 17, 2010 15  [[*Future directions]]  Robert P. Goldman committed Oct 19, 2010 16 17 18 19 ** TODO Fix "Pathname limitations" slide *** Needs structure *** Table? Something to make the before and after stand out *** Add the missing trailing slash in module pathnames to the slide...  20 21 22 23 24 25 26 27 ** TODO Missing extension hooks [[*Missing Extension Hooks]] *** Can we say what we did to fix this? Extend the published API? *** Possible bonus: Anything broadly interesting about the change? + Take home message about how to build your next API? + Ease/difficulty of changing API? OK if we don't have anything...  Francois-Rene Rideau committed Oct 17, 2010 28   Robert P. Goldman committed Oct 18, 2010 29 * COMMENT Possible theme for enhancements section  30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47  A number of things (returning values, handling cross-system dependencies properly) would be simplified if the ASDF plans were hierarchically (tree) structured, instead of being just flat sequences. Plans would have a flat, sequential form for execution, but there would be a superstructure that could accumulate values, etc. This would also help give operations on modules and systems a more intuitive semantics. Could also help let people write useful =:around= methods. Question: could we weave this into the concluding part of the talk. *POSSIBLY* --- propose not to do this unless it can be done reasonably clearly, and there may not be enough time remaining.  Robert P. Goldman committed Sep 30, 2010 48 49 50 51 52 53 54 55 56 57  * COMMENT Org-principles for this document ** Just like outline mode --- More "*" in header means deeper nesting ** Simple markup --- if you like! --- we can add this later *** /slashes/ for ital *** *stars* for bold *** =equal= signs for monospace ** Slide structure *** Single-star headings are sections *** Double-star headings are individual slides  Robert P. Goldman committed Sep 30, 2010 58 59 For this reason there are some things that look odd, where we have only a single sub-bullet under a major header.  Robert P. Goldman committed Sep 30, 2010 60 61 ** Outline proper starts in the following items  Robert P. Goldman committed Sep 30, 2010 62 63 64 * COMMENT Structure ** Suggest we have 15-18 slides ** I have 20 slides in the current draft  Robert P. Goldman committed Sep 30, 2010 65   Robert P. Goldman committed Sep 30, 2010 66 * Summary  Francois-Rene Rideau committed Oct 18, 2010 67 68 69 ** COMMENT Two slides here, for context ** ASDF  Francois-Rene Rideau committed Oct 19, 2010 70 *** *build system*: compile source files  Francois-Rene Rideau committed Oct 18, 2010 71 **** a bit like =make= for C or =ant= for Java  Francois-Rene Rideau committed Oct 19, 2010 72 *** *in image*: also load software  Francois-Rene Rideau committed Oct 18, 2010 73 **** totally unlike either =make= or =ant=  Robert P. Goldman committed Oct 18, 2010 74 **** maintain long-lived system state  Francois-Rene Rideau committed Oct 19, 2010 75 *** *declarative*: describe system dependencies  Francois-Rene Rideau committed Oct 18, 2010 76 **** not imperative instructions on how to build  Francois-Rene Rideau committed Oct 19, 2010 77 *** *specialized*: oriented toward Lisp software  Francois-Rene Rideau committed Oct 18, 2010 78 **** not for arbitrary tasks with dependencies  Francois-Rene Rideau committed Oct 19, 2010 79 80 81 *** *extensible*: can learn new classes of files **** not for arbitrary tasks with dependencies *** a key piece of CL *community* plumbing  Francois-Rene Rideau committed Oct 18, 2010 82 83 84 85  ** /Evolving/ ASDF *** Fixing up ASDF **** numerous issues: pathname limitations, difficult configuration, dependency bugs, fasl mess, portability, missing extension hooks  Robert P. Goldman committed Oct 18, 2010 86 *** ... not wrecking it in the process  Francois-Rene Rideau committed Oct 18, 2010 87 **** numerous existing users: some depend on obscure features, existing bugs, undocumented internals...  Robert P. Goldman committed Oct 19, 2010 88 *** How is evolving ASDF possible?  Francois-Rene Rideau committed Oct 18, 2010 89 90 91 92 93 *** We discovered interesting challenges **** technical: hot upgrade **** social: responsibility over upgrade, configuration *** Development principles **** some very well known, some less well known  Robert P. Goldman committed Sep 30, 2010 94 95  * What is ASDF?  Robert P. Goldman committed Sep 30, 2010 96 ** COMMENT 4 slides here  Robert P. Goldman committed Oct 19, 2010 97 ** Dominant build system for CL  Francois-Rene Rideau committed Oct 18, 2010 98 99 ***** COMMENT what is this thing below about? ***** COMMENT Not always going back to a clean slate like other tools  Robert P. Goldman committed Sep 30, 2010 100 101 102 103 104  *** Built on the shoulders of giants **** =MK-DEFSYSTEM= **** Symbolics and other proprietary =DEFSYSTEM= versions  Francois-Rene Rideau committed Oct 18, 2010 105 **** kmp's MIT AI Memo 801, rer's MIT AI TR 874.  Robert P. Goldman committed Sep 30, 2010 106   Robert P. Goldman committed Oct 14, 2010 107 *** Brilliant key idea establishes ASDF dominance  Robert P. Goldman committed Sep 30, 2010 108 109 110 111 112 113 114 115  **** Use =*load-truename*= to find system component files **** Suddenly, installing CL systems is easy + No more wrestling with logical pathnames + Especially since logical pathnames are insufficiently portable ** ASDF system definitions  Francois-Rene Rideau committed Oct 17, 2010 116 117 118 119 120 121 122 123 *** COLUMNS :B_columns: :PROPERTIES: :BEAMER_env: columns :END: **** 0.4 :B_column: :PROPERTIES: :BEAMER_env: column :END:  Robert P. Goldman committed Oct 14, 2010 124 125 126 127 128 129 + Contain /metadata/ + Contain /components/ + Contain /dependencies/ - external - internal  Francois-Rene Rideau committed Oct 17, 2010 130 131 132 133 134 135 136 137 138 139 **** 0.6 :B_column: :PROPERTIES: :BEAMER_env: column :END: #+BEAMER: \small #+BEGIN_SRC common-lisp (defsystem "foo" :version "1.0.1" :author "Fare and RPG" :depends-on (cl-ppcre)  Francois-Rene Rideau committed Oct 18, 2010 140 141  :components ((:file "blah") (:module "mod"  Francois-Rene Rideau committed Oct 17, 2010 142 143 144  :components ((:file "bar") (:file"baz")  Francois-Rene Rideau committed Oct 18, 2010 145  (:file "quux")))))  Francois-Rene Rideau committed Oct 17, 2010 146 147 #+END_SRC  Robert P. Goldman committed Sep 30, 2010 148 149 ** How does ASDF work?  Robert P. Goldman committed Oct 14, 2010 150 151 152 153 #+BEGIN_BEAMER \only<1>{\centerline{\includegraphics[width=10cm]{talk-images/find-system}}} \only<2>{\centerline{\includegraphics[width=10cm]{talk-images/defsystem}}} \only<3>{\centerline{\includegraphics[width=10cm]{talk-images/traverse}}}  Robert P. Goldman committed Oct 19, 2010 154 \only<4>{\centerline{\includegraphics[width=10cm]{talk-images/execute}}}  Robert P. Goldman committed Oct 14, 2010 155 156 157 158 159 160 161 162 163 164 165 166 167 #+END_BEAMER ** Three things I wish you knew about ASDF 1. ASDF doesn't just build: it makes a /plan/ to build, and then executes the plan 2. Operating on a module is not /wrapped around/ the operations on components + =:around= methods don't work for customization 3. You can tell ASDF about system versions + Say what the version of your system is + $x.y.z$, please + Please don't skimp on the high-order numbers or ASDF won't detect incompatibilities + Say what library versions it depends on  Francois-Rene Rideau committed Oct 19, 2010 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 * Fixing ASDF ** What was wrong with ASDF *** pathname limitations *** difficult configuration *** dependency bugs *** fasl mess *** portability *** missing extension hooks ** Pathname Limitations  Robert P. Goldman committed Oct 19, 2010 186 *** *Core idea:* #p"foo/bar" can never be portable. "foo/bar" can be.  Francois-Rene Rideau committed Oct 19, 2010 187 188 189 190 191 192 193 194 195  *** (:file "foo") ==> #p"foo.lisp" *** (:file "foo.bar") ==> before: #p"foo.bar" ASDF2: #p"foo.bar.lisp" *** (:file "foo-V1.2") ==> before: #p"foo-V1.2" ASDF2: #p"foo-V1.2.lisp" *** (:file "foo/bar") ==> before: non portable ASDF2: #p"foo/bar.lisp"  Robert P. Goldman committed Oct 19, 2010 196 **** (:file "foo-bar" :pathname #.(merge-pathnames (make-pathname :name "bar" :directory '(:relative "foo") :type "lisp" :defaults \*load-truename\*) \*load-truename\*))  Francois-Rene Rideau committed Oct 19, 2010 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228  *** (:static-file "README") ==> #p"README" *** (:file "README" :pathname #p"README") ==> before: #p"README.lisp" ASDF2: #p"README" ** Pathname fixes *** CL pathname FAIL: underspecified semantics means not portably usable **** Solution: provide a portable *abstraction* ***** parse strings into "relative pathnames with optionally specified type" *** ASDF FAIL: it was hard to get it actually right **** Solution: make it hard for the user to get it wrong *** ASDF FAIL: it was hard to know you were wrong **** Solution: better fail early for everyone than pass as working for some *** ASDF2 is much better, but still trips users **** Solution: keep improving code, error cases, documentation. ** Configuration Difficulties *** ** Output locations ** Dependency Bugs  Robert P. Goldman committed Oct 19, 2010 229 230 231 232 233 234 235 236  *** Changes within a module didn't cause recompilation of dependent components *** Changes on module dependencies didn't cause recompilation of module components *** Remember: systems are modules! *** Partially fixed --- see future directions  Francois-Rene Rideau committed Oct 19, 2010 237 238 239 240 241 242 243 244 245 246  ** Portability ** Missing Extension Hooks *** POIU (Parallel extension) needed to redefine some classes and methods *** ITA needed some extensions for printing compilation progress.  Robert P. Goldman committed Sep 30, 2010 247 248 * Hot-patching ASDF  Robert P. Goldman committed Sep 30, 2010 249 250 ** COMMENT 4 slides here  Robert P. Goldman committed Sep 30, 2010 251 252 253 254 255 ** Hot-patching ASDF *** Loading ASDF into a running CL image containing ASDF *** Why is this critical?  Robert P. Goldman committed Oct 14, 2010 256 257 258 259 260 261  **** If people can't load a new ASDF on top of an old one, they can never write portable code using new features **** /Unless/ all the implementations get together and /simultaneously/ update their packaged versions **** Hot-patching allows ASDF to evolve  Robert P. Goldman committed Sep 30, 2010 262 263 264  *** Why is this hard? **** We are replacing bits of ASDF /while it is running/  Robert P. Goldman committed Oct 14, 2010 265 **** While it is running /to build and load itself/  Robert P. Goldman committed Sep 30, 2010 266 267 **** COMMENT image of snake eating its own tail...  268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290  ** Hot-patching: how to do it *** Mostly easy: upgrading data, with =update-instance-for-redefined-class= **** "pull" model: everything sees the new schema. No "push" stages. *** Hard: changing a function's signature **** CLOS won't even let you do it. *** Rebinding (=fmakunbound=) vs Shadowing (=unintern= in all packages) **** No "rebind to transitional version that wraps the new version, then shadow" *** Harder: modified functions are in the continuation of =compile-file= or =load= **** Can't update continuation frames; they have no first-class names to be rebound. **** Worse if multithreaded: no atomicity  Robert P. Goldman committed Sep 30, 2010 291 * Configuration  Robert P. Goldman committed Sep 30, 2010 292 ** COMMENT 4 slides here  Robert P. Goldman committed Sep 30, 2010 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 ** Configuring ASDF *** Need to be able to find systems **** We can find system components from the system definition **** But we still need to find the system definitions! *** Need to place compiled files /Explain the need here/ ** Key issues are wrestling with CL pathnames * Best practices  Robert P. Goldman committed Sep 30, 2010 308 ** COMMENT 2 slides here  Robert P. Goldman committed Sep 30, 2010 309 310  * Lessons learned  Robert P. Goldman committed Sep 30, 2010 311 ** COMMENT 2 slides here  Robert P. Goldman committed Sep 30, 2010 312 313  * Future directions  Robert P. Goldman committed Sep 30, 2010 314 ** COMMENT 2 slides here  Robert P. Goldman committed Sep 30, 2010 315 316 317 318 319  ** Future directions *** There is still a lot of room for improvement with ASDF  Robert P. Goldman committed Oct 14, 2010 320 *** Bugs  Robert P. Goldman committed Sep 30, 2010 321   Francois-Rene Rideau committed Oct 17, 2010 322 323 324 *** New features *** Better documentation  Robert P. Goldman committed Oct 14, 2010 325 326   Francois-Rene Rideau committed Oct 17, 2010 327 ** Outstanding bug: across system dependencies do not properly trigger recompilations  Robert P. Goldman committed Sep 30, 2010 328   Francois-Rene Rideau committed Oct 17, 2010 329 + We /did/ fix a related bug --- module dependencies didn't work in ASDF 1  Robert P. Goldman committed Sep 30, 2010 330   Francois-Rene Rideau committed Oct 17, 2010 331 332 333 334 335 336 337 338 + Challenging to fix + Some think it's not a bug + Systems/libraries should provide stable APIs + Across sessions, ASDF doesn't have the information it needs + Not just FASLs newer than source files + What was the state of the image /when system FOO was compiled/? What was loaded? + Systems and modules are not components like others  Robert P. Goldman committed Oct 14, 2010 339   Robert P. Goldman committed Oct 17, 2010 340 341 342 343 344 345 346 347 348 349 ** Possible new features *** More declarative =DEFSYSTEM= **** Current procedural aspects make introspection difficult **** Need procedural code to invoke ASDF extensions **** Would require modifying how DEFSYSTEM is read  350 351 352 353 **** What about side-effect isolation? ***** See XCVB  Robert P. Goldman committed Oct 17, 2010 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 *** Make =TRAVERSE= part of the API **** Pro: Would allow cleaner extensions **** Con: Might lock in buggy behavior *** Conditional components **** E.g., only load this file if you are doing the =TEST-OP= *** More operations (e.g., =DOC-OP=) *** Return values **** =OPERATE= doesn't return a value --- bad for, e.g., testing **** Complicated by linear plans --- no hierarchical structure for collecting return values  372 373 *** COMMENT --- possible theme here --- having /hierarchical/ plans might make a lot of things easier.  374 375 ** What is still wrong with ASDF  376 *** Configuration is still harder than ideal  377   378 **** We tried to accommodate every previous extension variant.  379   380 **** Instead, should we have authoritatively standardized good defaults?  381   382 **** Or does our documentation just stink?  383 384 385 386 387  *** FASLs still not separate enough **** bit us with ASDF 2.006  388 389 390 391 392 *** The documentation *** Hard to extend **** At least partially due to the (lack of) documentation  393 394 395 396  *** ASDF2 is still not universally available  Robert P. Goldman committed Sep 30, 2010 397 398 * Conclusions  Robert P. Goldman committed Sep 30, 2010 399 400 ** COMMENT 1 slide here  Robert P. Goldman committed Oct 14, 2010 401 402 ** ASDF 2 is now available in stores near you...  Robert P. Goldman committed Oct 19, 2010 403 404 405  *** =http://common-lisp.net/project/asdf/=  Robert P. Goldman committed Oct 14, 2010 406 407 408 *** Download and load into your implementation of choice *** Bundled with an increasing number of implementations  Robert P. Goldman committed Sep 30, 2010 409   Robert P. Goldman committed Oct 14, 2010 410 411 412 413 **** COLUMNS :B_columns: :PROPERTIES: :BEAMER_env: columns :END:  Robert P. Goldman committed Sep 30, 2010 414   Robert P. Goldman committed Oct 14, 2010 415 416 417 418 ***** 0.5 :B_column: :PROPERTIES: :BEAMER_env: column :END:  Robert P. Goldman committed Sep 30, 2010 419   Robert P. Goldman committed Oct 14, 2010 420 ****** SBCL  Robert P. Goldman committed Sep 30, 2010 421   Robert P. Goldman committed Oct 14, 2010 422 ****** ABCL  Robert P. Goldman committed Sep 30, 2010 423   Robert P. Goldman committed Oct 14, 2010 424 ****** ECL  Robert P. Goldman committed Sep 30, 2010 425   Robert P. Goldman committed Oct 14, 2010 426 ****** SCL  Robert P. Goldman committed Sep 30, 2010 427   Robert P. Goldman committed Oct 14, 2010 428 ****** CMUCL  Robert P. Goldman committed Oct 14, 2010 429   Robert P. Goldman committed Oct 14, 2010 430 ****** CCL  Robert P. Goldman committed Oct 14, 2010 431   Robert P. Goldman committed Oct 14, 2010 432 433 434 435 ***** 0.5 :B_column: :PROPERTIES: :BEAMER_env: column :END:  Robert P. Goldman committed Oct 14, 2010 436   Robert P. Goldman committed Oct 14, 2010 437 ****** Yet to come (?)  Robert P. Goldman committed Oct 14, 2010 438   Francois-Rene Rideau committed Oct 18, 2010 439 440 ******* CLISP (discussed on the list)  Robert P. Goldman committed Oct 14, 2010 441 ******* ACL  Robert P. Goldman committed Oct 14, 2010 442   Robert P. Goldman committed Oct 14, 2010 443 ******* LispWorks  Robert P. Goldman committed Oct 14, 2010 444   Francois-Rene Rideau committed Oct 18, 2010 445 ******* Corman (no response)  Robert P. Goldman committed Sep 30, 2010 446   Robert P. Goldman committed Oct 17, 2010 447 448 ***** COMMENT end of columns  Robert P. Goldman committed Oct 14, 2010 449 *** =:asdf2= in =*features*=  Robert P. Goldman committed Sep 30, 2010 450   Robert P. Goldman committed Oct 14, 2010 451 452 453 ** Conclusions *** ASDF 2 needs new maintainers  Robert P. Goldman committed Sep 30, 2010 454 455 456 457  *** But please remember --- ASDF is the glue that holds the CL community together **** Be gentle with it!  Francois-Rene Rideau committed Oct 18, 2010 458 459 460  ** Thanks  Francois-Rene Rideau committed Oct 18, 2010 461 462 463 *** ASDF: Daniel Barlow, Christophe Rhodes, Gary King, ... **** A dozen contributors  Francois-Rene Rideau committed Oct 18, 2010 464 465 466 467 468 469 470  *** ASDF2: James Anderson, Juan-Jose Garcia-Ripoll *** Paper: Vadim Nasardinov, Ethan Schwartz, Scott McKay, Dan Barlow, anonymous reviewers *** All our users for testing --- and for building Lisp software!  Francois-Rene Rideau committed Oct 18, 2010 471 472 **** notable testing by Zach Beane, Samium Gromoff, Daniel Herring...  Francois-Rene Rideau committed Oct 19, 2010 473 ** Unupgradability of ASDF  Francois-Rene Rideau committed Oct 18, 2010 474   Robert P. Goldman committed Oct 19, 2010 475 *** To upgrade ASDF...  Francois-Rene Rideau committed Oct 18, 2010 476   Robert P. Goldman committed Oct 19, 2010 477 **** ASDF1: deep configuration before startup, in every implementation, every program  Francois-Rene Rideau committed Oct 18, 2010 478   Robert P. Goldman committed Oct 19, 2010 479 **** ASDF2: just install ASDF, no configuration unless non-standard location  Francois-Rene Rideau committed Oct 18, 2010 480   Francois-Rene Rideau committed Oct 19, 2010 481 **** you may rely on a feature...  Francois-Rene Rideau committed Oct 18, 2010 482   Francois-Rene Rideau committed Oct 19, 2010 483 ***** ASDF1: only if all supported implementations have upgraded  Francois-Rene Rideau committed Oct 18, 2010 484   Francois-Rene Rideau committed Oct 19, 2010 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 ***** ASDF2: anytime, ship recent ASDF if a feature isn't universally installed **** it brings value for an implementation to upgrade... ***** ASDF1: only after everybody else agreed to upgrade ***** ASDF2: anytime, or the implementation will lag behind **** After a given implementation upgrades... ***** ASDF1: more disuniformity in features available in installed base ***** ASDF2: more uniformity in features available in installed base ** First, fix upgrade incentives *** Dynamics of coordination **** In a stable process, each iteration provides more coordination than it requires *** Allowing for divergence creates an incentive towards convergence ** General Lessons Learned (Summary) **** ASDF is an in-image'' build system managing systems that are compiled and loaded in the current {\CL} image **** CL makes dynamic data upgrade extraordinarily easy. **** CL support for hot upgrade of code may exist but is anything but seamless **** the general problem with {\CL} is that its semantics are defined in terms of irreversible side-effects to global data structures in the current image **** Anything that can be provided as an extension should be provided as an extension and left out of the core **** We can't simply be transparent'' with respect to semantic discrepancies between underlying implementations; we must abstract those discrepancies away. **** have the person who knows write the configuration, not the person who doesn't **** good data structures and algorithms matter **** thou shalt tailor thy datastructures to the target problem **** Without the test suite, we'd be nowhere.