talk-outline.org 15.8 KB
Newer Older
Robert P. Goldman's avatar
Robert P. Goldman committed
1
2
3
#+LaTeX_CLASS: beamer
#+LaTeX_CLASS_OPTIONS: [presentation]
#+BEAMER_FRAME_LEVEL: 2
4
5
#+TITLE: Evolving ASDF
#+DATE: International Lisp Conference, 19 October 2010
6
7
#+AUTHOR: François-René Rideau \and Robert P. Goldman
#+BEAMER_HEADER_EXTRA: \usetheme{default}\usecolortheme{default} \subtitle{More cooperation, less coordination} \author{\begin{tabular} \ François-René Rideau \\ ITA Software \\ Cambridge, MA \end{tabular} \and \begin{tabular} \ Robert P. Goldman \\ SIFT, LLC \\ Minneapolis, MN \end{tabular}} \AtBeginSection[]{\begin{frame}<beamer> \frametitle{Outline} \tableofcontents[currentsection]\end{frame}}
Robert P. Goldman's avatar
Robert P. Goldman committed
8
9
10
#+COLUMNS: %45ITEM %10BEAMER_env(Env) %10BEAMER_envargs(Env Args) %4BEAMER_col(Col) %8BEAMER_extra(Extra)
#+PROPERTY: BEAMER_col_ALL 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 :ETC

11
12
* COMMENT To do items

Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
13
** DONE Center institutions under authors
Robert P. Goldman's avatar
Robert P. Goldman committed
14
** DONE Finish "Future Directions"
Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
15
  [[*Future directions]]
16
17
18
19
** TODO Fix "Pathname limitations" slide
*** Needs structure
*** Table?  Something to make the before and after stand out
*** Add the missing trailing slash in module pathnames to the slide...
20
21
22
23
24
25
26
27
** TODO Missing extension hooks
[[*Missing Extension Hooks]]
*** Can we say what we did to fix this? Extend the published API?
*** Possible bonus:  Anything broadly interesting about the change?
+ Take home message about how to build your next API?
+ Ease/difficulty of changing API?

OK if we don't have anything...
Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
28

29
* COMMENT Possible theme for enhancements section
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

A number of things (returning values, handling cross-system dependencies
properly) would be simplified if the ASDF plans were hierarchically (tree)
structured, instead of being just flat sequences.

Plans would have a flat, sequential form for execution, but there would be a
superstructure that could accumulate values, etc.

This would also help give operations on modules and systems a more intuitive
semantics.

Could also help let people write useful =:around= methods.

Question:  could we weave this into the concluding part of the talk.

*POSSIBLY* --- propose not to do this unless it can be done reasonably clearly,
 and there may not be enough time remaining.

Robert P. Goldman's avatar
Robert P. Goldman committed
48
49
50
51
52
53
54
55
56
57

* COMMENT Org-principles for this document
** Just like outline mode --- More "*" in header means deeper nesting
** Simple markup --- if you like! --- we can add this later
*** /slashes/ for ital
*** *stars* for bold
*** =equal= signs for monospace
** Slide structure
*** Single-star headings are sections
*** Double-star headings are individual slides
Robert P. Goldman's avatar
Robert P. Goldman committed
58
59
For this reason there are some things that look odd, where we have only a single
sub-bullet under a major header.
Robert P. Goldman's avatar
Robert P. Goldman committed
60
61
** Outline proper starts in the following items

Robert P. Goldman's avatar
Robert P. Goldman committed
62
63
64
* COMMENT Structure
** Suggest we have 15-18 slides
** I have 20 slides in the current draft
Robert P. Goldman's avatar
Robert P. Goldman committed
65

Robert P. Goldman's avatar
Robert P. Goldman committed
66
* Summary
67
68
69
** COMMENT Two slides here, for context

** ASDF
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
70
*** *build system*: compile source files
71
**** a bit like =make= for C or =ant= for Java
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
72
*** *in image*: also load software
73
**** totally unlike either =make= or =ant=
74
**** maintain long-lived system state
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
75
*** *declarative*: describe system dependencies
76
**** not imperative instructions on how to build
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
77
*** *specialized*: oriented toward Lisp software
78
**** not for arbitrary tasks with dependencies
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
79
80
81
*** *extensible*: can learn new classes of files
**** not for arbitrary tasks with dependencies
*** a key piece of CL *community* plumbing
82
83
84
85

** /Evolving/ ASDF
*** Fixing up ASDF
**** numerous issues: pathname limitations, difficult configuration, dependency bugs, fasl mess, portability, missing extension hooks
86
*** ... not wrecking it in the process
87
**** numerous existing users: some depend on obscure features, existing bugs, undocumented internals...
Robert P. Goldman's avatar
Robert P. Goldman committed
88
*** How is evolving ASDF possible?
89
90
91
92
93
*** We discovered interesting challenges
**** technical: hot upgrade
**** social: responsibility over upgrade, configuration
*** Development principles
**** some very well known, some less well known
Robert P. Goldman's avatar
Robert P. Goldman committed
94
95

* What is ASDF?
Robert P. Goldman's avatar
Robert P. Goldman committed
96
** COMMENT 4 slides here
Robert P. Goldman's avatar
Robert P. Goldman committed
97
** Dominant build system for CL
98
99
***** COMMENT what is this thing below about?
***** COMMENT Not always going back to a clean slate like other tools
Robert P. Goldman's avatar
Robert P. Goldman committed
100
101
102
103
104

*** Built on the shoulders of giants

**** =MK-DEFSYSTEM=
**** Symbolics and other proprietary =DEFSYSTEM= versions
105
**** kmp's MIT AI Memo 801, rer's MIT AI TR 874.
Robert P. Goldman's avatar
Robert P. Goldman committed
106

107
*** Brilliant key idea establishes ASDF dominance
Robert P. Goldman's avatar
Robert P. Goldman committed
108
109
110
111
112
113
114
115

**** Use =*load-truename*= to find system component files
**** Suddenly, installing CL systems is easy
+ No more wrestling with logical pathnames
+ Especially since logical pathnames are insufficiently portable

** ASDF system definitions

Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
116
117
118
119
120
121
122
123
*** COLUMNS                                                       :B_columns:
    :PROPERTIES:
    :BEAMER_env: columns
    :END:
**** 0.4                                                           :B_column:
     :PROPERTIES:
     :BEAMER_env: column
     :END:
124
125
126
127
128
129
+ Contain /metadata/
+ Contain /components/
+ Contain /dependencies/
  - external
  - internal

Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
130
131
132
133
134
135
136
137
138
139
**** 0.6                                                           :B_column:
     :PROPERTIES:
     :BEAMER_env: column
     :END:
#+BEAMER: \small
#+BEGIN_SRC common-lisp
(defsystem "foo"
  :version "1.0.1"
  :author "Fare and RPG"
  :depends-on (cl-ppcre)
140
141
  :components ((:file "blah")
               (:module "mod"
Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
142
143
144
                  :components
                   ((:file "bar")
                    (:file"baz")
145
                    (:file "quux")))))
Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
146
147
#+END_SRC

Robert P. Goldman's avatar
Robert P. Goldman committed
148
149
** How does ASDF work?

150
151
152
153
#+BEGIN_BEAMER
\only<1>{\centerline{\includegraphics[width=10cm]{talk-images/find-system}}}
\only<2>{\centerline{\includegraphics[width=10cm]{talk-images/defsystem}}}
\only<3>{\centerline{\includegraphics[width=10cm]{talk-images/traverse}}}
Robert P. Goldman's avatar
Robert P. Goldman committed
154
\only<4>{\centerline{\includegraphics[width=10cm]{talk-images/execute}}}
155
156
157
158
159
160
161
162
163
164
165
166
167
#+END_BEAMER

** Three things I wish you knew about ASDF

1. ASDF doesn't just build:  it makes a /plan/ to build, and then executes the plan
2. Operating on a module is not /wrapped around/ the operations on components
  + =:around= methods don't work for customization
3. You can tell ASDF about system versions
   + Say what the version of your system is
     + $x.y.z$, please
     + Please don't skimp on the high-order numbers or ASDF won't detect incompatibilities
   + Say what library versions it depends on

Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
* Fixing ASDF

** What was wrong with ASDF

*** pathname limitations

*** difficult configuration

*** dependency bugs

*** fasl mess

*** portability

*** missing extension hooks

184
185
* Pathname limitations

Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
186
187
** Pathname Limitations

188
*** *Core idea:* #p"foo/bar" can never be portable. "foo/bar" can be.
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
189
190
191
192
193
194
195
196
197

*** (:file "foo")       ==>  #p"foo.lisp"

*** (:file "foo.bar")   ==>  before: #p"foo.bar"    ASDF2: #p"foo.bar.lisp"

*** (:file "foo-V1.2")  ==>  before: #p"foo-V1.2"   ASDF2: #p"foo-V1.2.lisp"

*** (:file "foo/bar")   ==>  before: non portable   ASDF2: #p"foo/bar.lisp"

198
**** (:file "foo-bar" :pathname #.(merge-pathnames (make-pathname :name "bar" :directory '(:relative "foo") :type "lisp" :defaults \*load-truename\*) \*load-truename\*))
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225

*** (:static-file "README")   ==>  #p"README"

*** (:file "README" :pathname #p"README")   ==>  before: #p"README.lisp" ASDF2: #p"README"

** Pathname fixes

*** CL pathname FAIL: underspecified semantics means not portably usable

**** Solution: provide a portable *abstraction*

***** parse strings into "relative pathnames with optionally specified type"

*** ASDF FAIL: it was hard to get it actually right

**** Solution: make it hard for the user to get it wrong

*** ASDF FAIL: it was hard to know you were wrong

**** Solution: better fail early for everyone than pass as working for some

*** ASDF2 is much better, but still trips users

**** Solution: keep improving code, error cases, documentation.

** Configuration Difficulties

Robert P. Goldman's avatar
Robert P. Goldman committed
226
*** ASDF1 locates system definition files using a =*central-registry*=
227
228
229
230
231
232
233
234
235
236
237
238
239

*** But who configures it when?

**** .sbclrc, .ccl-init => onus on the user

**** =common-lisp-controller= => onus on the distributor

**** A program building script => onus on the release manager

*** ASDF2: decentralized source registry

**** environment variable, configuration files, defaults

Robert P. Goldman's avatar
Robert P. Goldman committed
240
*** *Each can specify what he knows, none need specify what he doesn't*
241
242
243
244

**** Essential principle for responsibility over configuration

*** COMMENT ASDF1: someone must manage integration of everything
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
245
246
247

** Output locations

248
249
250
251
252
253
254
255
*** CL isn't a single-implementation language.

**** Each version of each implementation has its own FASL format

*** Many ASDF1 extensions to move FASLs away (ABL, CLC, CL-launch)

**** Need to be magically loaded and configured

Robert P. Goldman's avatar
Robert P. Goldman committed
256
**** No consensus solution on where to put things
257
258
259
260
261
262
263

*** ASDF2: configurable output translations, sensible defaults

*** Unaddressed: further configuration discrepancies (e.g features, etc.)

**** ASDF won't ever do it. XCVB will (not yet).

Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
264
** Dependency Bugs
265
266
267
268
269
270
271
272

*** Changes within a module didn't cause recompilation of dependent components

*** Changes on module dependencies didn't cause recompilation of module components

*** Remember: systems are modules!

*** Partially fixed --- see future directions
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
273
274
275
276
277
278
279
280
281
282


** Portability

** Missing Extension Hooks

*** POIU (Parallel extension) needed to redefine some classes and methods

*** ITA needed some extensions for printing compilation progress.

Robert P. Goldman's avatar
Robert P. Goldman committed
283
284
* Hot-patching ASDF

Robert P. Goldman's avatar
Robert P. Goldman committed
285
286
** COMMENT 4 slides here

Robert P. Goldman's avatar
Robert P. Goldman committed
287
288
289
290
291
** Hot-patching ASDF

*** Loading ASDF into a running CL image containing ASDF

*** Why is this critical?
292
293
294
295
296
297

**** If people can't load a new ASDF on top of an old one, they can never write portable code using new features

**** /Unless/ all the implementations get together and /simultaneously/ update their packaged versions

**** Hot-patching allows ASDF to evolve
Robert P. Goldman's avatar
Robert P. Goldman committed
298
299
300

*** Why is this hard?
**** We are replacing bits of ASDF /while it is running/
301
**** While it is running /to build and load itself/
Robert P. Goldman's avatar
Robert P. Goldman committed
302
303
**** COMMENT image of snake eating its own tail...

304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326


** Hot-patching: how to do it

*** Mostly easy: upgrading data, with =update-instance-for-redefined-class=

**** "pull" model: everything sees the new schema. No "push" stages.

*** Hard: changing a function's signature

**** CLOS won't even let you do it.

*** Rebinding (=fmakunbound=) vs Shadowing (=unintern= in all packages)

**** No "rebind to transitional version that wraps the new version, then shadow"

*** Harder: modified functions are in the continuation of =compile-file= or =load=

**** Can't update continuation frames; they have no first-class names to be rebound.

**** Worse if multithreaded: no atomicity


Robert P. Goldman's avatar
Robert P. Goldman committed
327
* Configuration
Robert P. Goldman's avatar
Robert P. Goldman committed
328
** COMMENT 4 slides here
Robert P. Goldman's avatar
Robert P. Goldman committed
329
330
331
332
333
334
335
336
337
338
339
340
341
** Configuring ASDF

*** Need to be able to find systems

**** We can find system components from the system definition

**** But we still need to find the system definitions!

*** Need to place compiled files

/Explain the need here/

* Future directions
Robert P. Goldman's avatar
Robert P. Goldman committed
342
** COMMENT 2 slides here
Robert P. Goldman's avatar
Robert P. Goldman committed
343
344
345
346
347

** Future directions

*** There is still a lot of room for improvement with ASDF

348
*** Bugs
Robert P. Goldman's avatar
Robert P. Goldman committed
349

Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
350
351
352
*** New features

*** Better documentation
353
354


Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
355
** Outstanding bug: across system dependencies do not properly trigger recompilations
Robert P. Goldman's avatar
Robert P. Goldman committed
356

Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
357
+ We /did/ fix a related bug --- module dependencies didn't work in ASDF 1
Robert P. Goldman's avatar
Robert P. Goldman committed
358

Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
359
360
361
362
363
364
365
366
+ Challenging to fix
  +  Some think it's not a bug
    + Systems/libraries should provide stable APIs
  + Across sessions, ASDF doesn't have the information it needs
    + Not just FASLs newer than source files
    + What was the state of the image /when system FOO was compiled/?  What was
      loaded?
  + Systems and modules are not components like others
367

Robert P. Goldman's avatar
Robert P. Goldman committed
368
369
370
371
372
373
374
375
376
377
** Possible new features

*** More declarative =DEFSYSTEM=

**** Current procedural aspects make introspection difficult

**** Need procedural code to invoke ASDF extensions

**** Would require modifying how DEFSYSTEM is read

378
379
380
381
**** What about side-effect isolation?

***** See XCVB

Robert P. Goldman's avatar
Robert P. Goldman committed
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
*** Make =TRAVERSE= part of the API

**** Pro: Would allow cleaner extensions

**** Con: Might lock in buggy behavior

*** Conditional components

**** E.g., only load this file if you are doing the =TEST-OP=

*** More operations (e.g., =DOC-OP=)

*** Return values

**** =OPERATE= doesn't return a value --- bad for, e.g., testing

**** Complicated by linear plans --- no hierarchical structure for collecting return values

400
401
*** COMMENT --- possible theme here --- having /hierarchical/ plans might make a lot of things easier.

402
403
** What is still wrong with ASDF

404
*** Configuration is still harder than ideal
405

406
**** We tried to accommodate every previous extension variant.
407

408
**** Instead, should we have authoritatively standardized good defaults?
409

410
**** Or does our documentation just stink?
411
412
413
414
415

*** FASLs still not separate enough

**** bit us with ASDF 2.006

416
417
418
419
420
*** The documentation

*** Hard to extend

**** At least partially due to the (lack of) documentation
421
422
423
424

*** ASDF2 is still not universally available


Robert P. Goldman's avatar
Robert P. Goldman committed
425
426
* Conclusions

Robert P. Goldman's avatar
Robert P. Goldman committed
427
428
** COMMENT 1 slide here

429
430
** ASDF 2 is now available in stores near you...

431
432
433

*** =http://common-lisp.net/project/asdf/=

434
435
436
*** Download and load into your implementation of choice

*** Bundled with an increasing number of implementations
Robert P. Goldman's avatar
Robert P. Goldman committed
437

438
439
440
441
**** COLUMNS                                                      :B_columns:
     :PROPERTIES:
     :BEAMER_env: columns
     :END:
Robert P. Goldman's avatar
Robert P. Goldman committed
442

443
444
445
446
***** 0.5                                                          :B_column:
      :PROPERTIES:
      :BEAMER_env: column
      :END:
Robert P. Goldman's avatar
Robert P. Goldman committed
447

448
****** SBCL
Robert P. Goldman's avatar
Robert P. Goldman committed
449

450
****** ABCL
Robert P. Goldman's avatar
Robert P. Goldman committed
451

452
****** ECL
Robert P. Goldman's avatar
Robert P. Goldman committed
453

454
****** SCL
Robert P. Goldman's avatar
Robert P. Goldman committed
455

456
****** CMUCL
457

458
****** CCL
459

460
461
462
463
***** 0.5                                                          :B_column:
      :PROPERTIES:
      :BEAMER_env: column
      :END:
464

465
****** Yet to come (?)
466

467
468
******* CLISP (discussed on the list)

469
******* ACL
470

471
******* LispWorks
472

473
******* Corman (no response)
Robert P. Goldman's avatar
Robert P. Goldman committed
474

Robert P. Goldman's avatar
Robert P. Goldman committed
475
476
***** COMMENT end of columns

477
*** =:asdf2= in =*features*=
Robert P. Goldman's avatar
Robert P. Goldman committed
478

479
480
481
** Conclusions

*** ASDF 2 needs new maintainers
Robert P. Goldman's avatar
Robert P. Goldman committed
482
483
484
485

*** But please remember --- ASDF is the glue that holds the CL community together

**** Be gentle with it!
486
487
488

** Thanks

489
490
491
*** ASDF: Daniel Barlow, Christophe Rhodes, Gary King, ...

**** A dozen contributors
492
493
494
495
496
497
498

*** ASDF2: James Anderson, Juan-Jose Garcia-Ripoll

*** Paper: Vadim Nasardinov, Ethan Schwartz, Scott McKay, Dan Barlow, anonymous reviewers

*** All our users for testing --- and for building Lisp software!

499
500
**** notable testing by Zach Beane, Samium Gromoff, Daniel Herring...

Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
501
** Unupgradability of ASDF
502

503
*** To upgrade ASDF...
504

505
**** ASDF1: deep configuration before startup, in every implementation, every program
506

507
**** ASDF2: just install ASDF, no configuration unless non-standard location
508

Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
509
**** you may rely on a feature...
510

Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
511
***** ASDF1: only if all supported implementations have upgraded
512

Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
***** ASDF2: anytime, ship recent ASDF if a feature isn't universally installed

**** it brings value for an implementation to upgrade...

***** ASDF1: only after everybody else agreed to upgrade

***** ASDF2: anytime, or the implementation will lag behind

**** After a given implementation upgrades...

***** ASDF1: more disuniformity in features available in installed base

***** ASDF2: more uniformity in features available in installed base

** First, fix upgrade incentives


*** Dynamics of coordination

**** In a stable process, each iteration provides more coordination than it requires


*** Allowing for divergence creates an incentive towards convergence


** General Lessons Learned (Summary)

**** ASDF is an ``in-image'' build system managing systems that are compiled and loaded
     in the current {\CL} image
**** CL makes dynamic data upgrade extraordinarily easy.
**** CL support for hot upgrade of code may exist but is anything but seamless
**** the general problem with {\CL} is that its semantics are defined in terms
     of irreversible side-effects to global data structures in the current image

**** Anything that can be provided as an extension
     should be provided as an extension and left out of the core
**** We can't simply be ``transparent'' with respect to semantic discrepancies
     between underlying implementations; we must abstract those discrepancies away.

**** have the person who knows write the configuration, not the person who doesn't
**** good data structures and algorithms matter
**** thou shalt tailor thy datastructures to the target problem
**** Without the test suite, we'd be nowhere.