Let there be light

#+TITLE: Contributing

Contributions are welcome!

Note that I've tried to make the internal mechanisms extensible, but have not

yet exported the generic functions neccessary for other projects to extend the

behavior. They're not yet exported because I want to leave the option to

refactor things on the table until I know I have consumers. If you'd like those

GFs exported, just open an issue! After that point I'll make sure to not break

anything...

Copyright 2021 Eric Timmons <eric@timmons.dev>

Permission is hereby granted, free of charge, to any person obtaining a copy of

this software and associated documentation files (the "Software"), to deal in

the Software without restriction, including without limitation the rights to

use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies

of the Software, and to permit persons to whom the Software is furnished to do

so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all

copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR

IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,

FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE

AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER

LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,

OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE

SOFTWARE.

 No newline at end of file

#+TITLE: ASDF Package Inferred Components

ASDF's Package Inferred System class is pretty great. However, it has some

aspects that annoy me and this project is an attempt to clean those up, while

maintaining as many of the benefits as possible.

The best features of package inferred systems are:

1. Dependencies between files in the same project are automatically inferred.

2. Dependencies on external systems are declared very close to where they are

   used.

The worst features (IMHO) are:

1. Every single *file* in your project is a potential dependency for other

   systems. Because every file is turned into a system, people using your code

   can say ~(:depends-on :your-system/a/b/c)~ and then they suddenly become

   intimately tied to the implementation details of your project. Sure, you can

   say in your README "Only depend on ~:your-system~ or ~:your-system/a~", but

   why not expose only the systems you want?

2. There is no easy way to extract dependencies for a project. In order to do

   so, you must recurse into every sub-system. That is equivalent to recursing

   into every file and checking it for external dependencies.

3. Some features of ASDF dependency specification are unavailable. You cannot

   use ~:feature~ or ~:version~ dependencies in any of your inferred

   systems. ~:version~ dependencies are pretty easy to hoist to the top level

   system, but ~:feature~ dependencies are much trickier to get right.

4. It is difficult to include intraproject dependencies on anything other than

   Common Lisp source files. If you have a grovel-file from CFFI, there is no

   way to declaratively say that source file ~a/b/c.lisp~ depends on that

   grovel-file being operated on first.

5. It is difficult to introspect a system to determine its components. Similar

   to 2, you need to recurse into every subsystem/file to do so.

6. It is *required* that every CL source file has a corresponding package. This

   can help with modularity on large systems, but can be annoying on smaller

   systems. Plus some people plain don't like having that many packages around.

Most of these cons stem from the fact that the package-inferred-system class

creates an entirely new system for *every* lisp file in your project. This

project attempts to bring a little sanity to the inferred approach by inferring

*components* of a parent system.

* Notes

  + This currently requires a patched version of ASDF to work.

* Quickstart

** One package per system

   This example shows how to define a system ~z~ that uses fewer than one

   package per file. It has four source files: ~package~, ~a~, ~b~, and

   ~c~. ~a~, ~b~, and ~c~ all depend on ~package~. ~c~ also depends on both ~a~

   and ~b~. Additionally, the system depends on system ~x~, but let's assume

   all uses of ~x~ are localized to file ~b~.

   Without inferred components, your system definition would look like:

   #+begin_src common-lisp

     (defsystem #:z

       :components ((:file "package")

                    (:file "a" :depends-on ("package"))

                    (:file "b" :depends-on ("package"))

                    (:file "c" :depends-on ("package" "a" "b")))

       :depends-on (#:x))

   #+end_src

   With inferred components, your system definition would look like:

   #+begin_src common-lisp

     (defsystem #:z

       :defsystem-depends-on (#:asdf-inferred-components)

       :class "aic:inferred-components-system")

   #+end_src

   File ~a.lisp~ would start with:

   #+begin_src common-lisp

     (aic:define-component

         :depends-on (#:./package))

   #+end_src

   File ~b.lisp~ would start with:

   #+begin_src common-lisp

     (aic:define-component

         :depends-on (#:./package #:x))

   #+end_src

   File ~c.lisp~ would start with:

   #+begin_src common-lisp

     (aic:define-component

         :depends-on (#:./package #:./a #:./b))

   #+end_src

** One package per file

   This example shows the same as above, except each file also defines its own

   package. As such, we will also drop the ~package~ file.

   The system defintion would look like:

   #+begin_src common-lisp

     (defsystem #:z

       :defsystem-depends-on (#:asdf-inferred-components)

       :class "aic:inferred-components-system")

   #+end_src

   File ~a.lisp~ would start with:

   #+begin_src common-lisp

     (aic:define-package #:z/a

         (:use #:cl))

   #+end_src

   File ~b.lisp~ would start with:

   #+begin_src common-lisp

     (aic:define-package #:z/b

         (:use #:cl

               #:x))

   #+end_src

   File ~c.lisp~ would start with:

   #+begin_src common-lisp

     (aic:define-package #:z/c

         (:use #:cl

               #:./a

               #:./b))

   #+end_src

* How it works

  When the system object is being constructed, we descend into every folder

  under the system's pathname. A module is created for every directory, with

  the same name as the directory. Then, for every directory, we examine the

  files contained within it and create components for them.

  If the type of a file is recognized, then the file is examined for a

  component defining form. If the defining form exists, the options for the

  component (such as ~:depends-on~ and ~:if-feature~) are extracted and used to

  define the actual component object. This overrides all the options provided

  explicitly through ~defsystem~'s ~:components~ list (if any).

  If no component defining form is found, then the options for the file that

  are provided explicitly (via ~defsystem~'s ~:components~ list) are used.

  If the type of the file is not recognized and there is no explicit definition

  of the file, a component of type ~:static-file~ is created with no options.

* Recognized File Types

  This package recognizes the current file types:

  + ~lisp~, ~cl~, ~lsp~ :: Common Lisp source files

  + ~asd~ :: ASD definitions

  Any ASD definitions are ignored and *not* added as components. This helps

  handle the common case where the ~.asd~ file is in the same directory as the

  project's source files.

  Common Lisp source files are created using the ~:file~ component type.

* Component Defining Forms

** Common Lisp source files

   Component defining forms must be the first form in a file. Two forms are

   currently recognized:

   + ~(aic:define-component (&rest options &key depends-on component-type &allow-other-keys))~ :: This

     is a macro that expands to ~(progn)~ (i.e., it does nothing when the file

     is compiled or loaded).

     If ~component-type~ is provided, it must be something usable as the first

     form in the component definition. If not provided, it defaults to ~:file~.

     If ~depends-on~ is provided, it must be a list of dependency

     definitions. The provided list is processed before being handed off to

     ASDF's built in machinery.

     If the component name of any dependency starts with a ~.~ character, it is

     turned into an intrasystem dependency. To do so, the component name is

     parsed as a pathname relative to the file containing the defining

     form. Otherwise, the dependency is assumed to refer to a system.

     Every other option must be a keyword argument valid for explicitly

     defining a component.

   + ~(aic:define-package (package-name &rest options))~ :: This is a macro

     that is roughly equivalent to a ~aic:define-component~ form and a

     ~uiop:define-package~ form.

     The option ~(:component-def &rest plist)~ is used to define the options

     for creating the component. Every pair in ~plist~ is treated the same as

     if it were provided to ~aic:define-component~. There may be only one

     ~:component-def~ option.

     There may be any number of options of the form ~(:feature feature-expression &rest package-options)~.

     Every option in ~package-options~ is treated as described below, but its

     effects only happen if ~feature-expression~ evaluates to true (see

     ~uiop:featurep~).

     The remaining options are the package defining options. These options

     behave the same as if they were passed to ~uiop:define-package~. However,

     anywhere a package name is expected this macro also accepts a relative

     package name or a system name/package name pair.

     For example, if ~package-name~ is ~a/b/c~, the parent system is named ~a~,

     and ~options~ contains a form ~(:use #:../d/e)~, the package will use the

     package ~a/d/e~ and depend on the component ~("d" "e")~ in system ~a~.

     If ~options~ contains a form ~(:use (#:cl-z #:z))~, the package will use

     the package ~z~ and depend on the system ~cl-z~. If the first item in

     system/package pair is nil, no dependencies are extracted.

   While it would be possible to leave dependencies on other systems on each

   component (e.g., file ~a.lisp~ depends on system ~z~), doing so would likely

   break many tools and other features of ASDF. See

   [[https://gitlab.common-lisp.net/asdf/asdf/-/issues/68#note_7941]]. Therefore,

   any dependencies on systems are hoisted up to the parent system.

* Future Work

  1. Provide ability to specify component defining forms in file comments. This

     would be useful if someone wanted to use this project, but also make sure

     someone could load their system (by hand?) without having ASDF or AIC

     present.

     This could also lay the ground work for extending support beyond Common

     Lisp source files.

  2. Recognize files for CFFI's groveller. It's used enough that we should have

     first class support for it.

  3. Handle symlinks. I haven't tried it, but I wouldn't be surprised if things

     break horribly if there are symlinks within the source folders. But

     handling those isn't portable across OSes or implementations, so you're

     not using them, right?

  4. Add option to disable hoisting of external dependencies to the parent

     system.

;;;; ASDF-Inferred-Components System Definition

;;;;

;;;; This file is part of the asdf-inferred-components project. See README.org

;;;; and LICENSE for more information.

(defsystem #:asdf-inferred-components

  :version (:read-file-form "version.lisp-expr")

  :license "MIT"

  :pathname "src"

  :in-order-to ((test-op (load-op "asdf-inferred-components/test")))

  :perform (test-op (o c) (uiop:symbol-call :parachute :test :asdf-inferred-components-test))

  :components ((:file "package")

               (:file "file-wrappers" :depends-on ("package"))

               (:file "inferred-component-system" :depends-on ("package" "file-wrappers"))

               (:file "define-component" :depends-on ("package" "inferred-component-system"))

               (:file "define-package" :depends-on ("package" "inferred-component-system" "define-component"))))

(defsystem #:asdf-inferred-components/test

  :version (:read-file-form "version.lisp-expr")

  :license "MIT"

  :pathname "test"

  :components ((:file "package")

               (:file "aic-test-1")

               (:file "aic-test-2"))

  :depends-on (#:alexandria #:parachute))

;;; -*- mode: common-lisp -*-

(:api-version "0.3")

(:source "quicklisp"

 :type :ql-clpi

 :url "https://quicklisp.common-lisp-project-index.org/")

(:asd "asdf-inferred-components.asd")