Commit 19181e0b authored by Eric Timmons's avatar Eric Timmons
Browse files

Let there be light

parents
#+TITLE: Contributing
Contributions are welcome!
Note that I've tried to make the internal mechanisms extensible, but have not
yet exported the generic functions neccessary for other projects to extend the
behavior. They're not yet exported because I want to leave the option to
refactor things on the table until I know I have consumers. If you'd like those
GFs exported, just open an issue! After that point I'll make sure to not break
anything...
Copyright 2021 Eric Timmons <eric@timmons.dev>
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
\ No newline at end of file
#+TITLE: ASDF Package Inferred Components
ASDF's Package Inferred System class is pretty great. However, it has some
aspects that annoy me and this project is an attempt to clean those up, while
maintaining as many of the benefits as possible.
The best features of package inferred systems are:
1. Dependencies between files in the same project are automatically inferred.
2. Dependencies on external systems are declared very close to where they are
used.
The worst features (IMHO) are:
1. Every single *file* in your project is a potential dependency for other
systems. Because every file is turned into a system, people using your code
can say ~(:depends-on :your-system/a/b/c)~ and then they suddenly become
intimately tied to the implementation details of your project. Sure, you can
say in your README "Only depend on ~:your-system~ or ~:your-system/a~", but
why not expose only the systems you want?
2. There is no easy way to extract dependencies for a project. In order to do
so, you must recurse into every sub-system. That is equivalent to recursing
into every file and checking it for external dependencies.
3. Some features of ASDF dependency specification are unavailable. You cannot
use ~:feature~ or ~:version~ dependencies in any of your inferred
systems. ~:version~ dependencies are pretty easy to hoist to the top level
system, but ~:feature~ dependencies are much trickier to get right.
4. It is difficult to include intraproject dependencies on anything other than
Common Lisp source files. If you have a grovel-file from CFFI, there is no
way to declaratively say that source file ~a/b/c.lisp~ depends on that
grovel-file being operated on first.
5. It is difficult to introspect a system to determine its components. Similar
to 2, you need to recurse into every subsystem/file to do so.
6. It is *required* that every CL source file has a corresponding package. This
can help with modularity on large systems, but can be annoying on smaller
systems. Plus some people plain don't like having that many packages around.
Most of these cons stem from the fact that the package-inferred-system class
creates an entirely new system for *every* lisp file in your project. This
project attempts to bring a little sanity to the inferred approach by inferring
*components* of a parent system.
* Notes
+ This currently requires a patched version of ASDF to work.
* Quickstart
** One package per system
This example shows how to define a system ~z~ that uses fewer than one
package per file. It has four source files: ~package~, ~a~, ~b~, and
~c~. ~a~, ~b~, and ~c~ all depend on ~package~. ~c~ also depends on both ~a~
and ~b~. Additionally, the system depends on system ~x~, but let's assume
all uses of ~x~ are localized to file ~b~.
Without inferred components, your system definition would look like:
#+begin_src common-lisp
(defsystem #:z
:components ((:file "package")
(:file "a" :depends-on ("package"))
(:file "b" :depends-on ("package"))
(:file "c" :depends-on ("package" "a" "b")))
:depends-on (#:x))
#+end_src
With inferred components, your system definition would look like:
#+begin_src common-lisp
(defsystem #:z
:defsystem-depends-on (#:asdf-inferred-components)
:class "aic:inferred-components-system")
#+end_src
File ~a.lisp~ would start with:
#+begin_src common-lisp
(aic:define-component
:depends-on (#:./package))
#+end_src
File ~b.lisp~ would start with:
#+begin_src common-lisp
(aic:define-component
:depends-on (#:./package #:x))
#+end_src
File ~c.lisp~ would start with:
#+begin_src common-lisp
(aic:define-component
:depends-on (#:./package #:./a #:./b))
#+end_src
** One package per file
This example shows the same as above, except each file also defines its own
package. As such, we will also drop the ~package~ file.
The system defintion would look like:
#+begin_src common-lisp
(defsystem #:z
:defsystem-depends-on (#:asdf-inferred-components)
:class "aic:inferred-components-system")
#+end_src
File ~a.lisp~ would start with:
#+begin_src common-lisp
(aic:define-package #:z/a
(:use #:cl))
#+end_src
File ~b.lisp~ would start with:
#+begin_src common-lisp
(aic:define-package #:z/b
(:use #:cl
#:x))
#+end_src
File ~c.lisp~ would start with:
#+begin_src common-lisp
(aic:define-package #:z/c
(:use #:cl
#:./a
#:./b))
#+end_src
* How it works
When the system object is being constructed, we descend into every folder
under the system's pathname. A module is created for every directory, with
the same name as the directory. Then, for every directory, we examine the
files contained within it and create components for them.
If the type of a file is recognized, then the file is examined for a
component defining form. If the defining form exists, the options for the
component (such as ~:depends-on~ and ~:if-feature~) are extracted and used to
define the actual component object. This overrides all the options provided
explicitly through ~defsystem~'s ~:components~ list (if any).
If no component defining form is found, then the options for the file that
are provided explicitly (via ~defsystem~'s ~:components~ list) are used.
If the type of the file is not recognized and there is no explicit definition
of the file, a component of type ~:static-file~ is created with no options.
* Recognized File Types
This package recognizes the current file types:
+ ~lisp~, ~cl~, ~lsp~ :: Common Lisp source files
+ ~asd~ :: ASD definitions
Any ASD definitions are ignored and *not* added as components. This helps
handle the common case where the ~.asd~ file is in the same directory as the
project's source files.
Common Lisp source files are created using the ~:file~ component type.
* Component Defining Forms
** Common Lisp source files
Component defining forms must be the first form in a file. Two forms are
currently recognized:
+ ~(aic:define-component (&rest options &key depends-on component-type &allow-other-keys))~ :: This
is a macro that expands to ~(progn)~ (i.e., it does nothing when the file
is compiled or loaded).
If ~component-type~ is provided, it must be something usable as the first
form in the component definition. If not provided, it defaults to ~:file~.
If ~depends-on~ is provided, it must be a list of dependency
definitions. The provided list is processed before being handed off to
ASDF's built in machinery.
If the component name of any dependency starts with a ~.~ character, it is
turned into an intrasystem dependency. To do so, the component name is
parsed as a pathname relative to the file containing the defining
form. Otherwise, the dependency is assumed to refer to a system.
Every other option must be a keyword argument valid for explicitly
defining a component.
+ ~(aic:define-package (package-name &rest options))~ :: This is a macro
that is roughly equivalent to a ~aic:define-component~ form and a
~uiop:define-package~ form.
The option ~(:component-def &rest plist)~ is used to define the options
for creating the component. Every pair in ~plist~ is treated the same as
if it were provided to ~aic:define-component~. There may be only one
~:component-def~ option.
There may be any number of options of the form ~(:feature feature-expression &rest package-options)~.
Every option in ~package-options~ is treated as described below, but its
effects only happen if ~feature-expression~ evaluates to true (see
~uiop:featurep~).
The remaining options are the package defining options. These options
behave the same as if they were passed to ~uiop:define-package~. However,
anywhere a package name is expected this macro also accepts a relative
package name or a system name/package name pair.
For example, if ~package-name~ is ~a/b/c~, the parent system is named ~a~,
and ~options~ contains a form ~(:use #:../d/e)~, the package will use the
package ~a/d/e~ and depend on the component ~("d" "e")~ in system ~a~.
If ~options~ contains a form ~(:use (#:cl-z #:z))~, the package will use
the package ~z~ and depend on the system ~cl-z~. If the first item in
system/package pair is nil, no dependencies are extracted.
While it would be possible to leave dependencies on other systems on each
component (e.g., file ~a.lisp~ depends on system ~z~), doing so would likely
break many tools and other features of ASDF. See
[[https://gitlab.common-lisp.net/asdf/asdf/-/issues/68#note_7941]]. Therefore,
any dependencies on systems are hoisted up to the parent system.
* Future Work
1. Provide ability to specify component defining forms in file comments. This
would be useful if someone wanted to use this project, but also make sure
someone could load their system (by hand?) without having ASDF or AIC
present.
This could also lay the ground work for extending support beyond Common
Lisp source files.
2. Recognize files for CFFI's groveller. It's used enough that we should have
first class support for it.
3. Handle symlinks. I haven't tried it, but I wouldn't be surprised if things
break horribly if there are symlinks within the source folders. But
handling those isn't portable across OSes or implementations, so you're
not using them, right?
4. Add option to disable hoisting of external dependencies to the parent
system.
;;;; ASDF-Inferred-Components System Definition
;;;;
;;;; This file is part of the asdf-inferred-components project. See README.org
;;;; and LICENSE for more information.
(defsystem #:asdf-inferred-components
:version (:read-file-form "version.lisp-expr")
:license "MIT"
:pathname "src"
:in-order-to ((test-op (load-op "asdf-inferred-components/test")))
:perform (test-op (o c) (uiop:symbol-call :parachute :test :asdf-inferred-components-test))
:components ((:file "package")
(:file "file-wrappers" :depends-on ("package"))
(:file "inferred-component-system" :depends-on ("package" "file-wrappers"))
(:file "define-component" :depends-on ("package" "inferred-component-system"))
(:file "define-package" :depends-on ("package" "inferred-component-system" "define-component"))))
(defsystem #:asdf-inferred-components/test
:version (:read-file-form "version.lisp-expr")
:license "MIT"
:pathname "test"
:components ((:file "package")
(:file "aic-test-1")
(:file "aic-test-2"))
:depends-on (#:alexandria #:parachute))
;;; -*- mode: common-lisp -*-
(:api-version "0.3")
(:source "quicklisp"
:type :ql-clpi
:url "https://quicklisp.common-lisp-project-index.org/")
(:asd "asdf-inferred-components.asd")
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; This is autogenerated by CLPM. Do not edit by hand.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(:api-version "0.3")
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; BEGIN SOURCES
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
:sources
(:implicit-file :type :file-system :system-files
("asdf-inferred-components.asd"))
(:implicit-vcs :type :vcs :projects nil)
("quicklisp" :url "https://quicklisp.common-lisp-project-index.org/" :type
:ql-clpi)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; BEGIN REQUIREMENTS
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
:requirements
(:asd-file :name "asdf-inferred-components.asd")
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; BEGIN RELEASES
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
:releases
("alexandria" :version "2021-04-11" :source "quicklisp" :systems ("alexandria"))
("asdf-inferred-components.asd" :version :newest :source :implicit-file
:systems ("asdf-inferred-components" "asdf-inferred-components/test"))
("documentation-utils" :version "2019-07-11" :source "quicklisp" :systems
("documentation-utils"))
("form-fiddle" :version "2019-07-11" :source "quicklisp" :systems
("form-fiddle"))
("parachute" :version "2021-04-11" :source "quicklisp" :systems ("parachute"))
("trivial-indent" :version "2019-10-07" :source "quicklisp" :systems
("trivial-indent"))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; BEGIN REVERSE-DEPENDENCIES
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
:reverse-dependencies
("alexandria"
((:system :name "asdf-inferred-components/test") (:system :name "alexandria")))
("asdf-inferred-components.asd"
(t (:asd-file :name "asdf-inferred-components.asd")))
("documentation-utils"
((:system :name "parachute") (:system :name "documentation-utils"))
((:system :name "form-fiddle") (:system :name "documentation-utils")))
("form-fiddle" ((:system :name "parachute") (:system :name "form-fiddle")))
("parachute"
((:system :name "asdf-inferred-components/test") (:system :name "parachute")))
("trivial-indent"
((:system :name "documentation-utils") (:system :name "trivial-indent")))
;;;; ASDF-Inferred-Components System Definition
;;;;
;;;; This file is part of the asdf-inferred-components project. See README.org
;;;; and LICENSE for more information.
(in-package #:asdf-inferred-components)
(defmacro define-component (&rest args &key depends-on component-type &allow-other-keys)
"Define a new file component for a system. This (currently) expands
to (PROGN) (i.e., it does nothing when the file is compiled or loaded).
If COMPONENT-TYPE is provided, it must be something usable as the first form in
the component definition. If not provided, it defaults to :FILE.
If DEPENDS-ON is provided, it must be a list of dependency definitions. The
provided list is processed before being handed off to ASDF's built in
machinery.
If the component name of any dependency starts with a #\. character, it is
turned into an intrasystem dependency. To do so, the component name is parsed
as a pathname relative to the file containing the defining form. Otherwise, the
dependency is assumed to refer to a system.
Every other option must be a keyword argument valid for explicitly defining a
component."
(declare (ignore args depends-on component-type))
`(progn))
(defun resolve-dependency-component-name (parent-path id)
(if (uiop:string-prefix-p "." id)
(let ((relative-path (uiop:parse-unix-namestring id :want-relative t))
(path (reverse parent-path)))
(dolist (dir (rest (pathname-directory relative-path)))
(if (eql :back dir)
(pop path)
(push dir path)))
(push (pathname-name relative-path) path)
(reverse path))
id))
(defun parse-dependency-specifier (parent-path spec)
(etypecase spec
(string
(resolve-dependency-component-name parent-path spec))
(symbol
(resolve-dependency-component-name parent-path (asdf:coerce-name spec)))
(cons
(ecase (first spec)
(:version
(list* :version (parse-dependency-specifier parent-path (second spec))
(rest (rest spec))))
(:feature
(list :feature (second spec) (parse-dependency-specifier parent-path (third spec))))))))
(defmethod component-defining-list-to-plist (parent wrapper (operator (eql 'define-component)) args)
(destructuring-bind (&rest args &key depends-on &allow-other-keys) args
(let ((parent-path (component-path parent)))
(list* :depends-on (mapcar (lambda (x) (parse-dependency-specifier parent-path x)) depends-on)
(uiop:remove-plist-key :depends-on args)))))
;;;; ASDF-Inferred-Components System Definition
;;;;
;;;; This file is part of the asdf-inferred-components project. See README.org
;;;; and LICENSE for more information.
(in-package #:asdf-inferred-components)
(defun resolve-relative-package-name (base-package-designator relative-package-designator)
(let ((relative-name (string relative-package-designator))
(base-name (string base-package-designator)))
(if (uiop:string-prefix-p "." relative-name)
(let ((split-relative-name (uiop:split-string relative-name :separator "/"))
(name (rest (reverse (uiop:split-string base-name :separator "/")))))
(dolist (part split-relative-name)
(cond
((equal part "."))
((equal part "..")
(pop name))
(t
(push part name))))
(format nil "~{~A~^/~}" (reverse name)))
relative-package-designator)))
(defun parse-define-package-form (package-name options)
(let ((documentation nil)
(nicknames nil)
(use nil)
(shadow nil)
(shadowing-import-from nil)
(import-from nil)
(export nil)
(intern nil)
(recycle nil)
(mix nil)
(reexport nil)
(unintern nil)
(local-nicknames nil)
(component-def (find :component-def options :key #'first))
(no-feature (gensym)))
(labels
((guard-on-feature (feature thing)
(if (eql feature no-feature)
`',thing
`(when (uiop:featurep ',feature)
',thing)))
(process-package-spec (spec feature)
(if (listp spec)
(progn
(when (not (null (first spec)))
(push (if (eql feature no-feature)
(first spec)
`(:feature ,feature ,(first spec)))
(getf component-def :depends-on)))
(resolve-relative-package-name package-name (second spec)))
(progn
(unless (or (string-equal spec :cl)
(string-equal spec :common-lisp))
(push (if (eql feature no-feature)
spec
`(:feature ,feature ,spec))
(getf component-def :depends-on)))
(resolve-relative-package-name package-name spec))))
(process-package-list (packages feature)
(mapcar (lambda (p) (process-package-spec p feature)) packages))
(process-package-symbol-list (package-symbol-list feature)
(list* (process-package-spec (first package-symbol-list) feature)
(rest package-symbol-list)))
(process-symbol-package-list (symbol-package-list feature)
(list* (first symbol-package-list)
(mapcar (lambda (p) (process-package-spec p feature)) (rest symbol-package-list))))
(parse-option (option &optional (feature no-feature))
(destructuring-bind (kw &rest args) option
(ecase kw
(:nicknames
(push (guard-on-feature feature `(quote ,args)) nicknames))
(:documentation
(cond
(documentation (error "Can't define documentation twice"))
((or (not (stringp (first args))) (rest args)) (error "Bad documentation"))
(t (setf documentation (first args)))))
(:use
(push (guard-on-feature feature (process-package-list args feature))
use))
(:shadow
(push (guard-on-feature feature args)
shadow))
(:shadowing-import-from
(push (guard-on-feature feature (process-package-symbol-list args feature))
shadowing-import-from))
(:import-from
(push (guard-on-feature feature (process-package-symbol-list args feature))
import-from))
(:export
(push (guard-on-feature feature args)
export))
(:intern
(push (guard-on-feature feature args)
intern))
(:recycle
(push (guard-on-feature feature (process-package-list args feature))
recycle))
(:mix
(push (guard-on-feature feature (process-package-list args feature))
mix))
(:reexport
(push (guard-on-feature feature (process-package-list args feature))
reexport))
(:use-reexport
(push (guard-on-feature feature (process-package-list args feature))
use)
(push (guard-on-feature feature (process-package-list args feature))
reexport))
(:mix-reexport
(push (guard-on-feature feature (process-package-list args feature))
mix)
(push (guard-on-feature feature (process-package-list args feature))
reexport))
(:unintern
(push (guard-on-feature feature args)
unintern))
(:local-nicknames
(if (uiop:featurep :package-local-nicknames)
(push (guard-on-feature feature (mapcar (lambda (x) (process-symbol-package-list x feature))
args))
local-nicknames)
(error ":LOCAL-NICKNAMES option is not supported on this implementation.")))
(:feature
(mapc (lambda (o) (parse-option o (first args))) (rest args)))))))
(mapc #'parse-option (remove component-def options)))
(values
(list :nicknames `(append ,@(reverse nicknames))
:documentation documentation
:use (if use `(append ,@(reverse use)) '(quote (:common-lisp)))
:shadow `(append ,@(reverse shadow))
:shadowing-import-from `(remove nil (list ,@(reverse shadowing-import-from)))
:import-from `(remove nil (list ,@(reverse import-from)))
:export `(append ,@(reverse export))
:intern `(append ,@(reverse intern))
:recycle (if recycle `(append ,@(reverse recycle))
`(cons ',package-name (append ,@(reverse nicknames))))
:mix `(append ,@(reverse mix))
:reexport `(append ,@(reverse reexport))
:unintern `(append ,@(reverse unintern))
:local-nicknames `(append ,@(reverse local-nicknames)))
component-def)))
(defmacro define-package (package-name &rest options)
"Roughly equivalent to a AIC:DEFINE-COMPONENT form and a UIOP:DEFINE-PACKAGE form.
The option (:COMPONENT-DEF &REST PLIST) is used to define the options for
creating the component. Every pair in PLIST is treated the same as if it were
provided to AIC:DEFINE-COMPONENT. There may be only one :COMPONENT-DEF option.
There may be any number of options of the form (:FEATURE FEATURE-EXPRESSION &REST PACKAGE-OPTIONS).
Every option in PACKAGE-OPTIONS is treated as described below, but its effects
only happen if FEATURE-EXPRESSION evaluates to true (see UIOP:FEATUREP).
The remaining options are the package defining options. These options behave
the same as if they were passed to UIOP:DEFINE-PACKAGE. However, anywhere a
package name is expected this macro also accepts a relative package name or a
system name/package name pair.
For example, if PACKAGE-NAME is A/B/C, the parent system is named A, and
OPTIONS contains a form (:USE #:../D/E), the package will use the package A/D/E
and depend on the component (\"D\" \"E\") in system A.
If OPTIONS contains a form (:USE (#:CL-Z #:Z)), the package will use the
package Z and depend on the system CL-Z. If the first item in system/package
pair is NIL, no dependencies are extracted."
(let ((ensure-form
`(prog1
(funcall 'uiop:ensure-package ',package-name
,@(parse-define-package-form package-name options))
#+sbcl (setf (sb-impl::package-source-location (find-package ',package-name))