Commit 62de7e6a authored by Francois-Rene Rideau's avatar Francois-Rene Rideau
Browse files

(Nearly) camera-ready ILC'09 paper (hacked from rst2latex of submission).

More documentation updates on status and future plans.
Some worthless Lisp hacks (WIP to implement search path, general grains).
parent 0d1d3e3d
;;; Copyright (c) 2008-2009 ITA Software, Inc.
;;; Permission is hereby granted, free of charge, to any person
;;; obtaining a copy of this software and associated documentation
;;; files (the "Software"), to deal in the Software without
;;; restriction, including without limitation the rights to use, copy,
;;; modify, merge, publish, distribute, sublicense, and/or sell copies
;;; of the Software, and to permit persons to whom the Software is
;;; furnished to do so, subject to the following conditions:
;;; The above copyright notice and this permission notice shall be
;;; included in all copies or substantial portions of the Software.
......@@ -47,8 +47,8 @@ lisp-install:
## Janitoring
rm -f *.*fasl *.*fsl *.lib *.fas
rm -f xcvb doc/*.html
rm -f *.*fasl *.*fsl *.lib *.fas xcvb
cd doc ; rm -f *.html *.aux *.out *.bbl *.dvi *.log *.blg *.pdf
mrproper: clean
rm -f
......@@ -58,6 +58,14 @@ mrproper: clean
rst2html $< $@
doc/$(ILCPAPER).pdf: doc/$(ILCPAPER).tex doc/xcvb.bib
cd doc && pdflatex $(ILCPAPER) && bibtex $(ILCPAPER) && pdflatex $(ILCPAPER)
xpdf: doc/$(ILCPAPER).pdf
xpdf $<
doc: $(patsubst, %.html, $(wildcard doc/*.rest))
online-doc: doc
......@@ -70,4 +78,8 @@ push:
git show --pretty=oneline HEAD | head -1 | cut -d' ' -f1
.PHONY: all install lisp-install clean mrproper \
xpdf doc online-doc push show-current-revision
# To check out a particular revision: git fetch; git merge $commit
......@@ -191,10 +191,29 @@ called via a keyword in the appropriate place in a module declaration.
(1) A fullname is used as a unique name
A fullname is used as a unique name
to specify a component in the system.
Both modules and nodes have fullnames --
usually a node will have the same fullname as the module it was generated from
modulo a different file extension.
(2) For issues and TODO items, see
Random Remarks
On the choice of osicat vs cl-fad to access files,
I decided that fixing cl-fad is an uphill battle whereas extending osicat isn't:
to fix cl-fad, I need to fix it once per implementation,
with a growing number of implementations and interactions with many system-dependent variants;
to fix osicat, I need to fix it once per OS,
with a dwindling number of relevant OSes and implementation interactions already handled by cffi.
So, if we ever port XCVB to anything else than Unix,
the correct way will be to grow Windows (and JVM, Genera) support in OSICAT.
For issues and TODO items, see
......@@ -34,6 +34,11 @@ About XCVB
Bottom line:
XCVB isn't yet ready as a full replacement for ASDF.
But it is already working and useful
in the simple case with a single big project.
The basic functionality is already working:
you can either write an XCVB project or
automatically convert an existing ASDF project;
......@@ -85,18 +90,20 @@ You will need the following pieces of software to get XCVB running:
The original idea for what became XCVB came from discussions with James Knight
at ITA Software in 2007 and 2008.
The original idea for what became XCVB came from
discussions with James Knight at ITA Software in 2007.
XCVB was originally written by Spencer Brody under the supervision of
Francois-Rene Rideau from late May to mid August 2008.
The result was a working prototype,
but requiring some layer of hacks specific to the target on which XCVB was run
but requiring some layer of manual hacks
specific to the target on which XCVB was run
(an internal project at ITA Software).
Francois-Rene Rideau restarted work on it
and released an initial semi-usable prototype
in mid December 2008.
in mid December 2008, where the above hacks were automated away.
We hope to have a complete release with example and documentation
ready for showing at ILC'09 by March 2008.
......@@ -249,6 +256,9 @@ and the same side-effects need to be available at ``:load-toplevel``.
Therefore, any ``defun``, ``defvar`` or other side-effect used by a macro
needs be be enclosed in an ``(eval-when (:compile-toplevel :load-toplevel :execute) ...)``.
(Actually the ``:execute`` is not necessary in code you write manually,
but you probably want to include it in any macro that generates an ``eval-when``).
Further Limitations
......@@ -23,15 +23,17 @@ Future improvements planned for XCVB.
3.6 ASDF backend [V1]
3.7 Command-line interface [V3]
4 Features [V1]
4.1 Multiple BUILD.lisp files [V1]
4.2 Initialization and Finalization [V2]
4.3 Executables [V1]
4.4 Tests [V2]
4.5 Dependency grovelling [V2]
4.6 Revert [V1]
4.7 Bootstrap [V3]
4.8 Same source, multiple compilers [V4]
4.9 Bug hunting [V5]
4.1 Multiple BUILD.lisp files [V1]
4.2 Initialization and Finalization [V2]
4.3 Executables [V1]
4.4 Tests [V2]
4.5 Dependency grovelling [V2]
4.6 Revert [V1]
4.7 Bootstrap [V3]
4.8 Same source, multiple compilers [V4]
4.9 Correctly handling C libraries [V4]
4.10 Dynamic compilation [V4]
4.11 Bug hunting [V5]
5 The semantics of dependencies [V1]
5.1 FASL vs CFASL [V2]
5.2 Data files [V1]
......@@ -74,11 +76,13 @@ Future improvements planned for XCVB.
8.8 Push for First-class PCLSRing in Lisp compilers [V5]
8.9 Support dependencies for modules written in other languages [V4]
8.10 Mixing and Matching compilers [V5]
8.11 Constraint-based goal language [V6]
9 A Real Module System [V3]
9.1 Syntax Extensions [V3]
9.2 Automatic Package management [V5]
9.3 Better Namespace Management [V5]
9.4 Packaging and Distribution [V5]
9.5 More bibliography [V5]
Roadmap [V1]
......@@ -516,6 +520,32 @@ A very same executable could thus contain multiple versions of a same function
compiled with different options, to be used in different contexts.
Correctly handling C libraries [V4]
While objects may have to be compiled one way to be dynamically loaded into Lisp,
they may have to be compiled another way to be statically linked into the Lisp image.
XCVB will eventually have to know both ways, and do the right thing.
For dynamic objects, it should be able to have them installed in a place that
the Lisp image can find them when it needs them later.
For static objects, it should be able to recompile the Lisp runtime to include them.
This will require some synchronization with CFFI.
Dynamic compilation [V4]
XCVB should have a thin layer that enables to call it from an interactive environment,
and request the loading of some system.
The thin layer will collect the information about the running system
and about objects already loaded in it, as well as the additional systems required,
then invoke (with fork+exec) an xcvb process that will build any additional file
and print a form to be evaluated by the calling process so it may be suitably enhanced.
Bug hunting [V5]
......@@ -1183,6 +1213,12 @@ Of course, logical pathnames as defined by the CLHS are about unusable.
So offer to standardize something different for those virtual paths,
and the way that SLIME and/or the builtin debugger with interpret those paths.
jsnell suggests to look at what SLIME does -- presumably (for SBCL)
ask for extra information to be stored in the debug info with
a non-standard with-compilation-unit keyword. OR one might be able to do
sneaky stuff with logical-pathname-translations: ask to COMPILE-FILE
some logical-pathname for which an ad-hoc translation was created.
Push for better control of in-file source location [V4]
......@@ -1204,6 +1240,15 @@ tracker that only remembers the enclosing toplevel form wholly useless.
The PLT Scheme system has a good source tracking facility
that we may want to reuse of get inspiration from.
Even the C preprocessor has the crude line identification:
# 123 "baz.h" 2 3 4 ?
For SBCL, ask jsnell, tcr, nyef: the preprocessed file could expand to
things like #.(preprocessed-form 345) where the (preprocessed-form ...)
function would return expanded forms and subforms that each (as applicable)
already have source locations associated through something like
(setf (gethash EXPANDEDFORM *some-source-location-hash*)
Push for more determinism in Lisp compilers [V5]
......@@ -1269,6 +1314,17 @@ XCVB might grow some generic protocol to describe the steps required to
interface something to something else.
Constraint-based goal language [V6]
The build language of XCVB could in the most general case be
some constraint declaration language, and XCVB would use a constraint solver,
whereby you want to build a running system that has these final properties,
starting from an existing running system that has these initial properties.
XCVB could detect conformance of the specification with various known subsets
of the language that keep the constraint satisfaction easy.
A Real Module System [V3]
......@@ -1360,3 +1416,9 @@ See Ruby gems, Python eggs, Chicken eggs, Java beans.
When extending the system to include the management of packages for distribution,
be sure that you play well with whichever Operating System's packaging software:
More bibliography [V5]
......@@ -12,7 +12,7 @@
(trace write-makefile get-lisp-implementation)
;;; You may want to set parameters such as this one and others in specials.lisp
;;; You may want to set parameters such as these and others in specials.lisp
;(setf *lisp-implementation-type* :sbcl)
;(setf *lisp-executable-pathname* "/usr/local/bin/sbcl")
;(setf *lisp-image-pathname* "/path/to/my/monster-sbcl.core")
......@@ -20,8 +20,8 @@
;(setf *use-cfasls* t)
;;;; Not for stuff that's specific to the project.
;;; First, let's use the same setup file as used by cl-launch.
;;;; Now for stuff that's specific to the project.
;;; First, use the same setup file as when compiling with ASDF through cl-launch
;;; Importantly, setup.lisp will set the asdf:*central-registry*
(setf *lisp-setup-dependencies*
'((:asdf "cl-launch") ;; Required by setup.lisp
%-*- latex; coding: iso-8859-1; buffer-file-coding-system: iso-8859-1 -*-
% Faré: Latin1 encoding
\conferenceinfo{ILC '09}{2009-03-22, Cambridge, MA.}
\copyrightdata{[to be supplied]}
% The following lines are ignored unless 'preprint' option specified.
\titlebanner{Preprint of an ILC'09 paper}
\preprintfooter{This is a preprint}
\title{XCVB: Improving Modularity for Common Lisp}
%\subtitle{Subtitle Text, if any}
\authorinfo{François-René Rideau}
{ITA Software, Inc.}
\authorinfo{Spencer Brody}
{Brown University}
pdftitle={XCVB: Improving Modularity for Common Lisp},
pdfauthor={Francois-Rene Rideau {\textless}{\textgreater} and Spencer Brody {\textless}{\textgreater}}
%\thanks{This presentation describes work done at ITA Software, Inc.}
We present XCVB, a new open source system to build Common Lisp software.
XCVB notably features separate compilation.
This departs from the Common Lisp tradition
of compiling within a live world.
But separate compilation scales to very large systems,
and opens many possibilities for the future.
%term1, term2
%keyword1, keyword2
Despite plenty of advanced module systems
having been implemented in various Lisp dialects
(and languages from other families)
in the last 25 years,
the current state of the art for the Common Lisp community
hasn't improved much since 1980:
a central file defines the build graph,
and components are compiled and loaded
as successive side effects to the current Lisp world.
Our insight is to break the classic Lisp assumption of programming
in a single concrete Lisp world that is side-effected
as source code is sequentially compiled then loaded.
Instead, we propose a pure-functional approach of
building encapsulated components,
each compiled in its own isolated virtual Lisp world.
The traditional Lisp model may have been
ahead of its time in the 1970s and valid into the 1980s,
when memory and virtualization were expensive.
But it doesn't scale well to programming in the large in the 2000s,
with projects of tens of programmers with hundreds of files,
where the side-effects of processing various files
may interact in unexpected ways.
Our model, which is traditional for many other languages (like OCaml),
does scale to large projects and allows a concurrent and distributed build.
\section{Past Common Lisp build systems}
In the original DEFSYSTEM \cite{CHINE-NUAL} and its descendents,
a ``system'' is defined in a central file as
an acyclic graph of components
to be compiled and loaded into the current Lisp world,
arcs being dependencies.
The original DEFSYSTEM and
its once-popular portable descendent MK:DEFSYSTEM (from 1990 on)
required one to manually list all transitive dependencies of a component
in a statement separate from its definition
(see figure~\ref{dep:defsystem}).
The current state of the art and most popular descendent of DEFSYSTEM,
ASDF \cite{ASDF-Manual},
simplifies things tremendously
by automatically computing the transitive closure
of directly declared dependencies
from single statements defining both both component and direct dependencies
(see figure~\ref{dep:asdf}).
Such improvements and more had been suggested as early as 1984
but not available as a portable package until ASDF was created 2001.
ASDF also supports some limited form of extensibility using CLOS,
but for the purpose of this presentation,
we will focus on the core architecture of the build system.
Another DEFSYSTEM variant, mudballs, was recently created,
that seems to cleanup and simplify ASDF
while keeping the same general architecture.
(:module BAR ("BAR"))
(:compile-load BAR
\caption{Using DEFSYSTEM:
component {\tt bar} defined in two places in system file {\tt foo.system}
-- all transitive dependencies have to be listed explicitly.}
(:file "bar" :depends-on ("macros" "specials"))
\caption{Using ASDF:
component {\tt bar} defined somewhere in system file {\tt foo.asd}
-- transitive dependency on {\tt packages} may be skipped.}
(:depends-on ("macros" "specials")))
\caption{Using XCVB:
component {\tt bar} defined on top of module file {\tt bar.lisp}
-- transitive dependency on {\tt packages} is implicit}
XCVB, the eXtensible Component Verifier and Builder,
is a new open source system to build software written in Common Lisp.
XCVB most notably features separate compilation in isolated Lisp worlds.
The current implementation spawns a new Lisp process for each compilation.
In XCVB, dependency information is not declared in a central file,
but at the top of each component file
(see figure~\ref{dep:xcvb}).
In the following bullet-point presentation,
we briefly outline the reasons why separate compilation is important,
yet why it didn't happen for Common Lisp before.
We describe the state of the project and the benefits at hand.
We suggest the many future benefits that can be enabled by this design,
and we discuss the challenges that will have to be overcome
to achieve some of those benefits.
\section{XCVB yesterday}
What the XCVB prototype already brings as compared to ASDF.
\subsection{Goal: Separate Compilation}
Independent compilation of individual files.
Compute objects from source, just as in any modern language.
Proper staging of compile-time dependencies (macros compiled before they are used).
Semantics of a file fully encapsulated in its contents (+ contents of dependencies).
Incremental change-driven building and testing.
\subsection{Therefore: Dependencies must be declared locally}
Move dependencies away from centralized off-file meta-data.
Module import statement, just as in any modern language.
No more global recompilation (or subtle failure)
at the least change in centralized system definition.
No more subtle bugs to non-local change
in ordering of compile-time side-effects
(see figure \ref{asdf-break-down}).
Unlike ASDF, can incrementally track dependencies across systems.
Step 1: Initially working system.
;; bar provides base for baz
(:file "bar" :depends-on ("base"))
;; quux unrelatedly depends on base
(:file "quux" :depends-on ("base"))
;; baz also depends on base,
;; transitively provided by bar
(:file "baz" :depends-on ("bar"))
\vskip 1em
Step 2: after refactoring, {\tt bar} is simplified to not need {\tt base}
;; bar no longer depends on base
(:file "bar" :depends-on ("packages"))
;; quux happens to load base
;; before baz is compiled
(:file "quux" :depends-on ("base"))
;; baz really depends on base
;; but the missing dependency is not detected
(:file "baz" :depends-on ("bar"))
\vskip 1em
Step 3: {\tt quux} is also refactored to not need {\tt base}.
Now compilation of unrelated file {\tt baz} breaks
because its {\em implicit} dependency {\tt base} is not loaded anymore.
;; bar still does not depend on base
(:file "bar" :depends-on ("packages"))
;; quux no longer loads base
(:file "quux" :depends-on ("packages"))
;; despite a lack of related modifications,
;; baz now breaks inexplicably
;; at next clean compilation
(:file "baz" :depends-on ("bar"))
Of course, the lines that matter need not be consecutive
but separated by many unrelated lines,
the missing dependency may not be a direct dependency, but any transitive dependency,
and steps 2 and 3 may be separated by a lot of unrelated changes.
%First workaround: always make from clean
%(and lose any incremental building advantage).
%Second workaround: use a rigid serial dependency order
%(and lose most incremental building, accumulate coding scars).
%With XCVB, when you modify {\tt bar}, {\tt baz} automatically gets recompiled,
%and any missing dependency is immediately detected.
%No timebomb. No need for a workaround.
\caption{Break down of ASDF:
a mistake at step 2 creates a timebomb
that is only triggered
by an innocuous change much later at step 3.}
\subsection{Eager enforcement of dependencies}
Each file is built with none but the declared dependencies loaded.
Requires import discipline, just as in any modern language.
A bit slower, but much more robust: dependency bugs are detected early.
No more unmaintainable large manual dependency graphs
(or rigid unmaintained serial list of files as a workaround).
Allows eager incremental unit tests based on what has changed (beware: reflection).
\subsection{Current build backends}
XCVB computes the build graph, currently lets other software do the build.
{\tt Makefile}: integrate into a larger build, just as in any modern language.
ASDF: integrate into legacy ASDF builds.
More backends possible in the future:
{\tt OMake} \cite{OMake},
take over your build.
\subsection{Decoupling builder and buildee}
Protection from uncontrolled side-effects from buildee to builder.
Allows for integration with make as mentioned above.
Allows for cross-compilation from one compiler/architecture to a different one.
Allows for a feature-rich build system that needn't fit in one small file, yet.