Commit 9289cf3f authored by Francois-Rene Rideau's avatar Francois-Rene Rideau
Browse files

Update documentation of source-registry. Provide sensible default for inherit argument.

parent 1096d931
General Idea
============
The general idea is that in configuration files such as
/etc/common-lisp/source-registry.conf
~/.config/common-lisp/source-registry.conf
will be a SEXP in a trival domain-specific language
to specify a search path.
Backwards Compatibility
=======================
Now comes the issue of backwards compatibility with the current
asdf:*central-registry* that currently has to be manually configured
by users or supplemental layer (e.g. common-lisp-controller).
My proposal is that there would be a new (private) variable
common-lisp-configuration::*source-registry*
that would store a parsed version of the configuration
as an undocumented opaque data-structure
(except for the private use of ASDF and XCVB),
only accessible through API functions and said DSL.
One magic directive in that DSL, present by default in the configuration,
:default-registry
would hook into the implementation defaults, which would include
the searching of asdf:*central-registry* until it is eventually removed.
Uses of asdf:*central-registry* would be deprecated,
but still supported for many years to come for backwards compatibility.
The same DSL could be recognized by ASDF, XCVB, and any other future
build system for Common Lisp.
===========================
Common Lisp Source Registry
===========================
Alternatives I considered and rejected included:
This file specifies how build systems such as ASDF and XCVB
may be configured with respect to filesystem paths
where to search for Common Lisp source code.
1- Keep asdf:*central-registry* as the master with its current semantics,
and somehow the configuration parser expands the new configuration
language into a expanded series of directories of subdirectories to
lookup, pre-recursing through specified hierarchies. This is kludgy,
and leaves little space of future cleanups and extensions.
Configurations
==============
2- Keep asdf:*central-registry* remains the master but extend its semantics
in completely new ways, so that new kinds of entries may be implemented
as a recursive search, etc. This seems somewhat backwards.
Configurations specify paths where to find system files.
3- Completely remove asdf:*central-registry* and break backwards compatibility.
Hopefully this will happen in a few years after everyone migrate to
a better ASDF and/or to XCVB, but it would be very bad to do it now.
1- An application may explicitly initialize the source-registry
configuration using the `Configuration API`_ below,
in which case this takes precedence.
It may itself compute this configuration from the command-line,
from a script, from its own configuration file, etc.
4- Replace asdf:*central-registry* by a symbol-macro with appropriate magic
when you dereference it or setf it. Only the new variable with new
semantics is handled by the new search procedure.
2- The source registry will be configured from
the environment variable ``CL_SOURCE_REGISTRY`` if it exists.
3- The source registry will be configured from
user configuration file
``~/.config/common-lisp/source-registry.conf``
if it exists.
4- The source registry will be configured from
system configuration file
``/etc/common-lisp/source-registry.conf``
if it exists.
5- The source registry will be configured from a default configuration,
which allows for implementation-specific software to be searched.
(See below `Backward Compatibility`_).
Each of these configuration is specified as a SEXP
in a trival domain-specific language (defined below).
Additionally, a more shell-friendly syntax is available
for the environment variable (defined yet below).
Backward Compatibility
======================
For backward compatibility, ASDF will fall back to its old ways
if it fails to find a configuration for the source registry, or
if it fails to find a requested system in the configured source registry.
This new mechanism will therefore not affect you if you don't use it,
but will take precedence over the old mechanism if you do use it.
For the record, with the old mechanism, ASDF will first look
(on SBCL only) for a matching system in the implementation-specific
``contrib`` directory. Then (on all implementations) it will look
at your ``asdf:*central-registry*`` and search in the directories
specified there.
Configuration DSL
=================
Here is the grammar of the SEXP DSL I am considering for configuration:
Here is the grammar of the SEXP DSL for source-registry configuration:
;; A configuration is single SEXP starting with keyword :source-registry
;; followed by a list of directives.
......@@ -66,15 +71,15 @@ DIRECTIVE :=
(:directory DIRECTORY-PATHNAME-DESIGNATOR) |
;; add a directory hierarchy, recursing but excluding specified patterns
(:tree DIRECTORY-PATHNAME-DESIGNATOR &key exclude) |
(:tree DIRECTORY-PATHNAME-DESIGNATOR) |
;; override the default defaults for exclusion patterns
(:exclude-subdirectories PATTERN ...) |
(:exclude PATTERN ...) |
;; splice the parsed contents of another config file
(:include-configuration REGULAR-FILE-PATHNAME-DESIGNATOR) |
(:include REGULAR-FILE-PATHNAME-DESIGNATOR) |
;; Your configuration expression MUST have contain exactly one of these:
;; Your configuration expression MUST contain exactly one of either of these:
(:inherit-configuration) | ; splices contents of inherited configuration
(:ignore-inherited-configuration) ; drop contents of inherited configuration
......@@ -82,99 +87,141 @@ DIRECTIVE :=
(:default-registry)
PATTERN := a string without wildcards, that will be matched exactly
against the name of a subdirectory.
against the name of a any subdirectory in the directory component of a path.
e.g. "_darcs" will match #p"/foo/bar/_darcs/src/bar.asd"
Configuration Files
===================
Shell-friendly syntax for configuration
=======================================
Following a suggestion by Stelian, the configuration should be read in
this order, each configuration containing an expression that may extend
or override the previous configuration:
file /etc/common-lisp/source-registry.conf
file ~/.config/common-lisp/source-registry.conf
environment variable CL_SOURCE_REGISTRY
some implementation- or application- specific command-line argument.
When considering environment variable ``CL_SOURCE_REGISTRY``
ASDF will skip to next configuration if it's an empty string.
It will ``READ`` the string as a SEXP in the DSL
if it begins with a paren ``(``
and it will be interpreted much like ``TEXINPUTS``
as ``:`` (colon) separated list of paths, where
* each entry is a directory to add to the search path.
* if the entry ends with a double slash ``//``
then it instead indicates a tree in the subdirectories of which to recurse.
* if the entry is the empty string (which may only appear once),
then it indicates that the inherited configuration should be spliced there.
Search Algorithm
================
Equivalently (since the DSL doesn't allow for uncontrolled side-effects),
the outermost configuration only would be parsed,
and recursing to previous configurations would only happen
if and when specified.
In case that isn't clear, the semantics of the configuration is that
when searching for a system of a given name, directives are processed in order.
This also leaves the question of whether environment variable and
command-line arguments should take the same SEXP syntax (to be READ),
or whether they should have some more shell-friendly textual
representation of the same information. Allowing for a different
representation is probably more work for marginally more feature,
and I don't recommend it. So, in the end, the same SEXP syntax
for shell variables and command-line arguments.
When looking in a directory, if the system is found, the search succeeds,
otherwise it continues.
When looking in a tree, if one system is found, the search succeeds.
If multiple systems are found, the consequences are unspecified:
the search may succeed with any of the found systems,
or an error may be raised.
ASDF currently returns the first system found,
XCVB currently raised an error.
If none is found, the search continues.
Exclude statements specify patterns of subdirectories the systems of which
to ignore. Typically you don't want to use copies of files kept by such
version control systems as Darcs.
Include statements cause the search to recurse with the path specifications
from the file specified.
An inherit-configuration statement cause the search to recurse with the path
specifications from the next configuration (see section Configurations_ above).
Caching Results
===============
The implementation is allowed to either eagerly compute the information
from the configurations and file system, or to lazily re-compute it every time,
or to cache any part of it as it goes.
To explicitly flush any information cached by the system, use the API below.
Configuration API
=================
This API is exported from package COMMON-LISP-CONFIGURATION.
The specified functions are exported from your build system's package.
Thus for ASDF the corresponding functions are in package ASDF,
and for XCVB the corresponding functions are in package XCVB.
(initialize-source-registry)
will read the configuration and initialize all internal variables,
and return the new configuration.
will read the configuration and initialize all internal variables.
(clear-source-registry)
undoes any initialization. You might want to call that before you
undoes any source registry configuration
and clears any cache for the search algorithm.
You might want to call that before you
dump an image that would be resumed with a different configuration,
and return an empty configuration.
Also will have a hook that allows clients will use to clear any cache
that depends on this configuration.
Note that this does not include clearing information about systems defined
in the current image, only about where to look for systems not yet defined.
(ensure-source-registry)
checks an initial variable to see whether the state is initialized
or cleared. In the former case, return current configuration;
in the latter, initialize.
ASDF will call this function at the start of (asdf:find-system).
(process-source-registry X &optional inherit)
checks whether a source registry has been initialized.
If not, initialize it.
This function will be called before any attempt to find a system
in the source registry.
If your application wants to override the provided defaults,
it will have to use the below function process-source-registry.
(process-source-registry X &key inherit collect)
If X is a CONS, parse it as a SEXP in the configuration DSL,
and extend or override inheritted configuration.
If X is a STRING, first parse it into a SEXP with READ
(Alternate proposal: parse some shell-friendly text representation).
The inheritted configuration is provided in optional argument inherit,
itself a function that returns the previous configuration, with NIL
designating the default of #'ensure-source-registry-configuration.
Internally, initialize-source-registry-configuration can use this
with a series of functions for inheritted configuration.
If X is a STRING, first parse it into a SEXP as for the CL_SOURCE_REGISTRY
environment variable (see above) then process it.
If X is a PATHNAME, read the file as a single SEXP and process it.
The inheritted configuration is provided in keyword argument inherit,
itself a list of functions that take inherit and collect keyword arguments
and defaulting to a list of functions that implements the default behavior.
Search Algorithm
================
Future
======
* When a system is searched for, entries are processed in order.
If this mechanism is successful, in the future, we may declare
``asdf:*central-registry*`` obsolete and eventually remove it.
Any hook into implementation-specific search mechanisms will by then
have been integrated in the ``:default-configuration`` which everyone
should either explicitly use or implicit inherit. Some shell syntax
for it should probably be added somehow.
* If a given entry has exactly one match, the search stops successfully.
If a given entry contains no match, it is skipped.
If a given entry contains multiple matches, an error is thrown.
But we're not there yet. For now, let's see how practical this new
source-registry is.
* This later case does not change the semantics of ASDF in the case
where no recursion takes place, and ensures no undetected insanity
happens in the case where recursion is specified. XCVB has tested
this model, with success I believe.
* When an entry is first processed, the implementation may cache
the contents of the directory (i.e. all files that may match anything.)
until the cache is explicitly flushed (see below).
Rejected ideas
==============
Alternatives I considered and rejected included:
Cache flushing
==============
1- Keep asdf:*central-registry* as the master with its current semantics,
and somehow the configuration parser expands the new configuration
language into a expanded series of directories of subdirectories to
lookup, pre-recursing through specified hierarchies. This is kludgy,
and leaves little space of future cleanups and extensions.
* The cache is flushed when function
(asdf:clear-system-search-cache)
is called.
2- Keep asdf:*central-registry* remains the master but extend its semantics
in completely new ways, so that new kinds of entries may be implemented
as a recursive search, etc. This seems somewhat backwards.
* When the configuration is (re)loaded, the cache is flushed.
3- Completely remove asdf:*central-registry* and break backwards compatibility.
Hopefully this will happen in a few years after everyone migrate to
a better ASDF and/or to XCVB, but it would be very bad to do it now.
4- Replace asdf:*central-registry* by a symbol-macro with appropriate magic
when you dereference it or setf it. Only the new variable with new
semantics is handled by the new search procedure.
Questioned Niceties
===================
I've been suggested the below features, but have rejected them,
for the sake of keeping ASDF no more complex than strictly necessary.
......@@ -192,19 +239,19 @@ for the sake of keeping ASDF no more complex than strictly necessary.
i.e. uses of USER-HOMEDIR-PATHNAME and $SBCL_HOME
Hopefully, these are already superseded by the :default-registry
* We may allow a shell-friendly colon-separated text-based syntax
for environment variables and command-line arguments.
If we only accept directories, not files, then all provided strings should
be considered with implicit / at the end before being parsed by
parse-namestring or such. An explicit // (as in TEXINPUTS) or /**
(as in zsh or some (non-standard) CL pathname extensions, but
requiring painful quotes from the shell) could specify recursion.
* Using the shell-unfriendly syntax /** instead of // to specify recursion
down a filesystem tree in the environment variable.
It isn't that Lisp friendly either.
Additional Notes
================
Credits
=======
Thanks a lot to Stelian Ionescu for the initial idea.
Thanks to Rommel Martinez for the initial implementation attempt.
With this specification, we can share configuration between ASDF and XCVB,
and maintain reasonable backwards compatibility in ASDF.
All bad design ideas and implementation bugs are to mine, not theirs.
But so are good design ideas and elegant implementation tricks.
-- Francois-Rene Rideau <fare@tunes.org>, Sun, 17 Jan 2010 15:53:31 -0500
-- Francois-Rene Rideau <fare@tunes.org>, Mon, 18 Jan 2010 14:12:57 -0500
......@@ -2121,9 +2121,8 @@ applied by the plain `*source-to-target-mappings*`."
(pushnew 'contrib-sysdef-search *system-definition-search-functions*))
;;;; -----------------------------------------------------------------
;;;; TODO: Read Configuration.
;;;; Source Registry Configuration. See README.source-registry
;;;; See https://bugs.launchpad.net/asdf/+bug/485918
;;;; TODO: add protocols for re-searching a loaded system in the registry,
;;;; for invalidating registry entries, etc.
;;;; See https://bugs.launchpad.net/asdf/+bug/485687
......@@ -2154,8 +2153,6 @@ applied by the plain `*source-to-target-mappings*`."
"Undoes any initialization of the source registry.
You might want to call that before you dump an image that would be resumed
with a different configuration, so the configuration would be re-read then."
;; TODO: in the future, have a hook that clients will use to clear any cache
;; that depends on this configuration.
(setf *source-registry* '())
(values))
......@@ -2242,9 +2239,7 @@ with a different configuration, so the configuration would be re-read then."
:for i = (or (position #\: string :start start) end) :do
(let ((s (subseq string start i)))
(cond
((equal "" s) ; empty element
nil)
((equal "!" s) ; inherit
((equal "" s) ; empty element: inherit
(when inherit
(error "only one inherited configuration allowed: ~S" string))
(setf inherit t)
......@@ -2297,15 +2292,21 @@ with a different configuration, so the configuration would be re-read then."
nil)
(defgeneric process-source-registry (spec &key inherit collect))
(defmethod process-source-registry ((pathname pathname) &key inherit collect)
(defmethod process-source-registry ((pathname pathname) &key
(inherit *default-source-registries*)
collect)
(if (probe-file pathname)
(process-source-registry (validate-source-registry-file pathname)
:inherit inherit :collect collect)
(inherit-source-registry inherit :collect collect)))
(defmethod process-source-registry ((string string) &key inherit collect)
(defmethod process-source-registry ((string string) &key
(inherit *default-source-registries*)
collect)
(process-source-registry (parse-source-registry-string string)
:inherit inherit :collect collect))
(defmethod process-source-registry ((x null) &key inherit collect)
(defmethod process-source-registry ((x null) &key
(inherit *default-source-registries*)
collect)
(inherit-source-registry inherit :collect collect))
(defun make-collector ()
......@@ -2313,7 +2314,9 @@ with a different configuration, so the configuration would be re-read then."
(values (lambda (x) (push x acc))
(lambda () (reverse acc)))))
(defmethod process-source-registry ((form cons) &key inherit collect)
(defmethod process-source-registry ((form cons) &key
(inherit *default-source-registries*)
collect)
(multiple-value-bind (collect result)
(if collect
(values collect (constantly nil))
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment