talk-outline.org 14.9 KB
Newer Older
Robert P. Goldman's avatar
Robert P. Goldman committed
1
2
3
#+LaTeX_CLASS: beamer
#+LaTeX_CLASS_OPTIONS: [presentation]
#+BEAMER_FRAME_LEVEL: 2
4
5
#+TITLE: Evolving ASDF
#+DATE: International Lisp Conference, 19 October 2010
6
7
#+AUTHOR: François-René Rideau \and Robert P. Goldman
#+BEAMER_HEADER_EXTRA: \usetheme{default}\usecolortheme{default} \subtitle{More cooperation, less coordination} \author{\begin{tabular} \ François-René Rideau \\ ITA Software \\ Cambridge, MA \end{tabular} \and \begin{tabular} \ Robert P. Goldman \\ SIFT, LLC \\ Minneapolis, MN \end{tabular}} \AtBeginSection[]{\begin{frame}<beamer> \frametitle{Outline} \tableofcontents[currentsection]\end{frame}}
Robert P. Goldman's avatar
Robert P. Goldman committed
8
9
10
#+COLUMNS: %45ITEM %10BEAMER_env(Env) %10BEAMER_envargs(Env Args) %4BEAMER_col(Col) %8BEAMER_extra(Extra)
#+PROPERTY: BEAMER_col_ALL 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 :ETC

11
12
* COMMENT To do items

Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
13
** DONE Center institutions under authors
Robert P. Goldman's avatar
Robert P. Goldman committed
14
** DONE Finish "Future Directions"
Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
15
  [[*Future directions]]
16
17
18
19
** TODO Fix "Pathname limitations" slide
*** Needs structure
*** Table?  Something to make the before and after stand out
*** Add the missing trailing slash in module pathnames to the slide...
20
21
22
23
24
25
26
27
** TODO Missing extension hooks
[[*Missing Extension Hooks]]
*** Can we say what we did to fix this? Extend the published API?
*** Possible bonus:  Anything broadly interesting about the change?
+ Take home message about how to build your next API?
+ Ease/difficulty of changing API?

OK if we don't have anything...
Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
28

29
* COMMENT Possible theme for enhancements section
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

A number of things (returning values, handling cross-system dependencies
properly) would be simplified if the ASDF plans were hierarchically (tree)
structured, instead of being just flat sequences.

Plans would have a flat, sequential form for execution, but there would be a
superstructure that could accumulate values, etc.

This would also help give operations on modules and systems a more intuitive
semantics.

Could also help let people write useful =:around= methods.

Question:  could we weave this into the concluding part of the talk.

*POSSIBLY* --- propose not to do this unless it can be done reasonably clearly,
 and there may not be enough time remaining.

Robert P. Goldman's avatar
Robert P. Goldman committed
48
49
50
51
52
53
54
55
56
57

* COMMENT Org-principles for this document
** Just like outline mode --- More "*" in header means deeper nesting
** Simple markup --- if you like! --- we can add this later
*** /slashes/ for ital
*** *stars* for bold
*** =equal= signs for monospace
** Slide structure
*** Single-star headings are sections
*** Double-star headings are individual slides
Robert P. Goldman's avatar
Robert P. Goldman committed
58
59
For this reason there are some things that look odd, where we have only a single
sub-bullet under a major header.
Robert P. Goldman's avatar
Robert P. Goldman committed
60
61
** Outline proper starts in the following items

Robert P. Goldman's avatar
Robert P. Goldman committed
62
63
64
* COMMENT Structure
** Suggest we have 15-18 slides
** I have 20 slides in the current draft
Robert P. Goldman's avatar
Robert P. Goldman committed
65

Robert P. Goldman's avatar
Robert P. Goldman committed
66
* Summary
67
68
69
** COMMENT Two slides here, for context

** ASDF
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
70
*** *build system*: compile source files
71
**** a bit like =make= for C or =ant= for Java
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
72
*** *in image*: also load software
73
**** totally unlike either =make= or =ant=
74
**** maintain long-lived system state
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
75
*** *declarative*: describe system dependencies
76
**** not imperative instructions on how to build
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
77
*** *specialized*: oriented toward Lisp software
78
**** not for arbitrary tasks with dependencies
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
79
80
81
*** *extensible*: can learn new classes of files
**** not for arbitrary tasks with dependencies
*** a key piece of CL *community* plumbing
82
83
84
85

** /Evolving/ ASDF
*** Fixing up ASDF
**** numerous issues: pathname limitations, difficult configuration, dependency bugs, fasl mess, portability, missing extension hooks
86
*** ... not wrecking it in the process
87
**** numerous existing users: some depend on obscure features, existing bugs, undocumented internals...
Robert P. Goldman's avatar
Robert P. Goldman committed
88
*** How is evolving ASDF possible?
89
90
91
92
93
*** We discovered interesting challenges
**** technical: hot upgrade
**** social: responsibility over upgrade, configuration
*** Development principles
**** some very well known, some less well known
Robert P. Goldman's avatar
Robert P. Goldman committed
94
95

* What is ASDF?
Robert P. Goldman's avatar
Robert P. Goldman committed
96
** COMMENT 4 slides here
Robert P. Goldman's avatar
Robert P. Goldman committed
97
** Dominant build system for CL
98
99
***** COMMENT what is this thing below about?
***** COMMENT Not always going back to a clean slate like other tools
Robert P. Goldman's avatar
Robert P. Goldman committed
100
101
102
103
104

*** Built on the shoulders of giants

**** =MK-DEFSYSTEM=
**** Symbolics and other proprietary =DEFSYSTEM= versions
105
**** kmp's MIT AI Memo 801, rer's MIT AI TR 874.
Robert P. Goldman's avatar
Robert P. Goldman committed
106

107
*** Brilliant key idea establishes ASDF dominance
Robert P. Goldman's avatar
Robert P. Goldman committed
108
109
110
111
112
113
114
115

**** Use =*load-truename*= to find system component files
**** Suddenly, installing CL systems is easy
+ No more wrestling with logical pathnames
+ Especially since logical pathnames are insufficiently portable

** ASDF system definitions

Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
116
117
118
119
120
121
122
123
*** COLUMNS                                                       :B_columns:
    :PROPERTIES:
    :BEAMER_env: columns
    :END:
**** 0.4                                                           :B_column:
     :PROPERTIES:
     :BEAMER_env: column
     :END:
124
125
126
127
128
129
+ Contain /metadata/
+ Contain /components/
+ Contain /dependencies/
  - external
  - internal

Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
130
131
132
133
134
135
136
137
138
139
**** 0.6                                                           :B_column:
     :PROPERTIES:
     :BEAMER_env: column
     :END:
#+BEAMER: \small
#+BEGIN_SRC common-lisp
(defsystem "foo"
  :version "1.0.1"
  :author "Fare and RPG"
  :depends-on (cl-ppcre)
140
141
  :components ((:file "blah")
               (:module "mod"
Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
142
143
144
                  :components
                   ((:file "bar")
                    (:file"baz")
145
                    (:file "quux")))))
Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
146
147
#+END_SRC

Robert P. Goldman's avatar
Robert P. Goldman committed
148
149
** How does ASDF work?

150
151
152
153
#+BEGIN_BEAMER
\only<1>{\centerline{\includegraphics[width=10cm]{talk-images/find-system}}}
\only<2>{\centerline{\includegraphics[width=10cm]{talk-images/defsystem}}}
\only<3>{\centerline{\includegraphics[width=10cm]{talk-images/traverse}}}
Robert P. Goldman's avatar
Robert P. Goldman committed
154
\only<4>{\centerline{\includegraphics[width=10cm]{talk-images/execute}}}
155
156
157
158
159
160
161
162
163
164
165
166
167
#+END_BEAMER

** Three things I wish you knew about ASDF

1. ASDF doesn't just build:  it makes a /plan/ to build, and then executes the plan
2. Operating on a module is not /wrapped around/ the operations on components
  + =:around= methods don't work for customization
3. You can tell ASDF about system versions
   + Say what the version of your system is
     + $x.y.z$, please
     + Please don't skimp on the high-order numbers or ASDF won't detect incompatibilities
   + Say what library versions it depends on

Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
* Fixing ASDF

** What was wrong with ASDF

*** pathname limitations

*** difficult configuration

*** dependency bugs

*** fasl mess

*** portability

*** missing extension hooks

** Pathname Limitations

186
*** *Core idea:* #p"foo/bar" can never be portable. "foo/bar" can be.
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
187
188
189
190
191
192
193
194
195

*** (:file "foo")       ==>  #p"foo.lisp"

*** (:file "foo.bar")   ==>  before: #p"foo.bar"    ASDF2: #p"foo.bar.lisp"

*** (:file "foo-V1.2")  ==>  before: #p"foo-V1.2"   ASDF2: #p"foo-V1.2.lisp"

*** (:file "foo/bar")   ==>  before: non portable   ASDF2: #p"foo/bar.lisp"

196
**** (:file "foo-bar" :pathname #.(merge-pathnames (make-pathname :name "bar" :directory '(:relative "foo") :type "lisp" :defaults \*load-truename\*) \*load-truename\*))
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228

*** (:static-file "README")   ==>  #p"README"

*** (:file "README" :pathname #p"README")   ==>  before: #p"README.lisp" ASDF2: #p"README"

** Pathname fixes

*** CL pathname FAIL: underspecified semantics means not portably usable

**** Solution: provide a portable *abstraction*

***** parse strings into "relative pathnames with optionally specified type"

*** ASDF FAIL: it was hard to get it actually right

**** Solution: make it hard for the user to get it wrong

*** ASDF FAIL: it was hard to know you were wrong

**** Solution: better fail early for everyone than pass as working for some

*** ASDF2 is much better, but still trips users

**** Solution: keep improving code, error cases, documentation.

** Configuration Difficulties

*** 

** Output locations

** Dependency Bugs
229
230
231
232
233
234
235
236

*** Changes within a module didn't cause recompilation of dependent components

*** Changes on module dependencies didn't cause recompilation of module components

*** Remember: systems are modules!

*** Partially fixed --- see future directions
Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
237
238
239
240
241
242
243
244
245
246


** Portability

** Missing Extension Hooks

*** POIU (Parallel extension) needed to redefine some classes and methods

*** ITA needed some extensions for printing compilation progress.

Robert P. Goldman's avatar
Robert P. Goldman committed
247
248
* Hot-patching ASDF

Robert P. Goldman's avatar
Robert P. Goldman committed
249
250
** COMMENT 4 slides here

Robert P. Goldman's avatar
Robert P. Goldman committed
251
252
253
254
255
** Hot-patching ASDF

*** Loading ASDF into a running CL image containing ASDF

*** Why is this critical?
256
257
258
259
260
261

**** If people can't load a new ASDF on top of an old one, they can never write portable code using new features

**** /Unless/ all the implementations get together and /simultaneously/ update their packaged versions

**** Hot-patching allows ASDF to evolve
Robert P. Goldman's avatar
Robert P. Goldman committed
262
263
264

*** Why is this hard?
**** We are replacing bits of ASDF /while it is running/
265
**** While it is running /to build and load itself/
Robert P. Goldman's avatar
Robert P. Goldman committed
266
267
**** COMMENT image of snake eating its own tail...

268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290


** Hot-patching: how to do it

*** Mostly easy: upgrading data, with =update-instance-for-redefined-class=

**** "pull" model: everything sees the new schema. No "push" stages.

*** Hard: changing a function's signature

**** CLOS won't even let you do it.

*** Rebinding (=fmakunbound=) vs Shadowing (=unintern= in all packages)

**** No "rebind to transitional version that wraps the new version, then shadow"

*** Harder: modified functions are in the continuation of =compile-file= or =load=

**** Can't update continuation frames; they have no first-class names to be rebound.

**** Worse if multithreaded: no atomicity


Robert P. Goldman's avatar
Robert P. Goldman committed
291
* Configuration
Robert P. Goldman's avatar
Robert P. Goldman committed
292
** COMMENT 4 slides here
Robert P. Goldman's avatar
Robert P. Goldman committed
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
** Configuring ASDF

*** Need to be able to find systems

**** We can find system components from the system definition

**** But we still need to find the system definitions!

*** Need to place compiled files

/Explain the need here/

** Key issues are wrestling with CL pathnames

* Best practices
Robert P. Goldman's avatar
Robert P. Goldman committed
308
** COMMENT 2 slides here
Robert P. Goldman's avatar
Robert P. Goldman committed
309
310

* Lessons learned
Robert P. Goldman's avatar
Robert P. Goldman committed
311
** COMMENT 2 slides here
Robert P. Goldman's avatar
Robert P. Goldman committed
312
313

* Future directions
Robert P. Goldman's avatar
Robert P. Goldman committed
314
** COMMENT 2 slides here
Robert P. Goldman's avatar
Robert P. Goldman committed
315
316
317
318
319

** Future directions

*** There is still a lot of room for improvement with ASDF

320
*** Bugs
Robert P. Goldman's avatar
Robert P. Goldman committed
321

Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
322
323
324
*** New features

*** Better documentation
325
326


Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
327
** Outstanding bug: across system dependencies do not properly trigger recompilations
Robert P. Goldman's avatar
Robert P. Goldman committed
328

Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
329
+ We /did/ fix a related bug --- module dependencies didn't work in ASDF 1
Robert P. Goldman's avatar
Robert P. Goldman committed
330

Francois-Rene Rideau's avatar
Merge    
Francois-Rene Rideau committed
331
332
333
334
335
336
337
338
+ Challenging to fix
  +  Some think it's not a bug
    + Systems/libraries should provide stable APIs
  + Across sessions, ASDF doesn't have the information it needs
    + Not just FASLs newer than source files
    + What was the state of the image /when system FOO was compiled/?  What was
      loaded?
  + Systems and modules are not components like others
339

Robert P. Goldman's avatar
Robert P. Goldman committed
340
341
342
343
344
345
346
347
348
349
** Possible new features

*** More declarative =DEFSYSTEM=

**** Current procedural aspects make introspection difficult

**** Need procedural code to invoke ASDF extensions

**** Would require modifying how DEFSYSTEM is read

350
351
352
353
**** What about side-effect isolation?

***** See XCVB

Robert P. Goldman's avatar
Robert P. Goldman committed
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
*** Make =TRAVERSE= part of the API

**** Pro: Would allow cleaner extensions

**** Con: Might lock in buggy behavior

*** Conditional components

**** E.g., only load this file if you are doing the =TEST-OP=

*** More operations (e.g., =DOC-OP=)

*** Return values

**** =OPERATE= doesn't return a value --- bad for, e.g., testing

**** Complicated by linear plans --- no hierarchical structure for collecting return values

372
373
*** COMMENT --- possible theme here --- having /hierarchical/ plans might make a lot of things easier.

374
375
** What is still wrong with ASDF

376
*** Configuration is still harder than ideal
377

378
**** We tried to accommodate every previous extension variant.
379

380
**** Instead, should we have authoritatively standardized good defaults?
381

382
**** Or does our documentation just stink?
383
384
385
386
387

*** FASLs still not separate enough

**** bit us with ASDF 2.006

388
389
390
391
392
*** The documentation

*** Hard to extend

**** At least partially due to the (lack of) documentation
393
394
395
396

*** ASDF2 is still not universally available


Robert P. Goldman's avatar
Robert P. Goldman committed
397
398
* Conclusions

Robert P. Goldman's avatar
Robert P. Goldman committed
399
400
** COMMENT 1 slide here

401
402
** ASDF 2 is now available in stores near you...

403
404
405

*** =http://common-lisp.net/project/asdf/=

406
407
408
*** Download and load into your implementation of choice

*** Bundled with an increasing number of implementations
Robert P. Goldman's avatar
Robert P. Goldman committed
409

410
411
412
413
**** COLUMNS                                                      :B_columns:
     :PROPERTIES:
     :BEAMER_env: columns
     :END:
Robert P. Goldman's avatar
Robert P. Goldman committed
414

415
416
417
418
***** 0.5                                                          :B_column:
      :PROPERTIES:
      :BEAMER_env: column
      :END:
Robert P. Goldman's avatar
Robert P. Goldman committed
419

420
****** SBCL
Robert P. Goldman's avatar
Robert P. Goldman committed
421

422
****** ABCL
Robert P. Goldman's avatar
Robert P. Goldman committed
423

424
****** ECL
Robert P. Goldman's avatar
Robert P. Goldman committed
425

426
****** SCL
Robert P. Goldman's avatar
Robert P. Goldman committed
427

428
****** CMUCL
429

430
****** CCL
431

432
433
434
435
***** 0.5                                                          :B_column:
      :PROPERTIES:
      :BEAMER_env: column
      :END:
436

437
****** Yet to come (?)
438

439
440
******* CLISP (discussed on the list)

441
******* ACL
442

443
******* LispWorks
444

445
******* Corman (no response)
Robert P. Goldman's avatar
Robert P. Goldman committed
446

Robert P. Goldman's avatar
Robert P. Goldman committed
447
448
***** COMMENT end of columns

449
*** =:asdf2= in =*features*=
Robert P. Goldman's avatar
Robert P. Goldman committed
450

451
452
453
** Conclusions

*** ASDF 2 needs new maintainers
Robert P. Goldman's avatar
Robert P. Goldman committed
454
455
456
457

*** But please remember --- ASDF is the glue that holds the CL community together

**** Be gentle with it!
458
459
460

** Thanks

461
462
463
*** ASDF: Daniel Barlow, Christophe Rhodes, Gary King, ...

**** A dozen contributors
464
465
466
467
468
469
470

*** ASDF2: James Anderson, Juan-Jose Garcia-Ripoll

*** Paper: Vadim Nasardinov, Ethan Schwartz, Scott McKay, Dan Barlow, anonymous reviewers

*** All our users for testing --- and for building Lisp software!

471
472
**** notable testing by Zach Beane, Samium Gromoff, Daniel Herring...

Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
473
** Unupgradability of ASDF
474

475
*** To upgrade ASDF...
476

477
**** ASDF1: deep configuration before startup, in every implementation, every program
478

479
**** ASDF2: just install ASDF, no configuration unless non-standard location
480

Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
481
**** you may rely on a feature...
482

Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
483
***** ASDF1: only if all supported implementations have upgraded
484

Francois-Rene Rideau's avatar
Francois-Rene Rideau committed
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
***** ASDF2: anytime, ship recent ASDF if a feature isn't universally installed

**** it brings value for an implementation to upgrade...

***** ASDF1: only after everybody else agreed to upgrade

***** ASDF2: anytime, or the implementation will lag behind

**** After a given implementation upgrades...

***** ASDF1: more disuniformity in features available in installed base

***** ASDF2: more uniformity in features available in installed base

** First, fix upgrade incentives


*** Dynamics of coordination

**** In a stable process, each iteration provides more coordination than it requires


*** Allowing for divergence creates an incentive towards convergence


** General Lessons Learned (Summary)

**** ASDF is an ``in-image'' build system managing systems that are compiled and loaded
     in the current {\CL} image
**** CL makes dynamic data upgrade extraordinarily easy.
**** CL support for hot upgrade of code may exist but is anything but seamless
**** the general problem with {\CL} is that its semantics are defined in terms
     of irreversible side-effects to global data structures in the current image

**** Anything that can be provided as an extension
     should be provided as an extension and left out of the core
**** We can't simply be ``transparent'' with respect to semantic discrepancies
     between underlying implementations; we must abstract those discrepancies away.

**** have the person who knows write the configuration, not the person who doesn't
**** good data structures and algorithms matter
**** thou shalt tailor thy datastructures to the target problem
**** Without the test suite, we'd be nowhere.