Commit 78d7f05a authored by Eric Timmons's avatar Eric Timmons
Browse files

Ready to build and start hammering on it in my daily workflow

parent f3fd77ab
root = true
[*.lisp]
indent_style = space
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
[*.asd]
indent_style = space
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
.dir-locals-2.el
/build
*.fasl
image: daewok/sbcl:latest
build:
stage: build
script:
- sbcl --no-sysinit --no-userinit --non-interactive --load build.lisp --eval "(clpm-build:build)"
#+TITLE: CLPM Changelog
* v0.0.1 - 2018-10-13
Initial release.
Copyright 2018 Eric Timmons
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
\ No newline at end of file
#+TITLE: Common Lisp Package Manager - CLPM
#+AUTHOR: Eric Timmons
#+EMAIL: etimmons@mit.edu
#+OPTIONS: email:t toc:2 num:nil
**WARNING**: This is ALPHA quality. It is being published right now so I can
start testing it and hammering at it in my normal work flow. There are definite
bugs and it will eat your code.
* Description
CLPM is a package manager for Common Lisp. It supports installing either
versioned releases of packages or development releases straight from source
control. Additionally, it allows versioned dependencies to be specified for
individual projects you are hacking on, allowing you to create project
specific contexts.
It consists of two major pieces. First is a CLI program that is responsible
for all the heavy lifting of fetching and unpacking releases in the correct
place, finding systems, and managing project specific environments. The second
is a small client library written in portable Common Lisp that interfaces with
ASDF and calls the CLI program as necessary to find and install missing
systems.
* Motivation
CLPM is far from the only package manager available for Common Lisp[fn:1], but
it makes very different assumptions and design choices than the other
available solutions. We try to point those out in this section.
+ Minimize footprint in client image :: CLPM handles most of the heavy work of
parsing source lists, downloading packages, installing packages, and
finding installed packages in a separate process. This keeps the size of
any image using CLPM small and enables many of the other benefits
discussed later.
+ Use existing libraries where possible :: Because CLPM is never loaded into a
client image, there are no issues with version conflicts or
incompatibilities between third party libraries used by CLPM and the
client image. This means that existing libraries can be leveraged where
it makes sense and we don't need to reimplement things inside CLPM itself.
+ Support HTTPS :: There are Lisp packages that support requests over HTTPS
and we can use them without fear of bloating or causing
issues in the client image.
+ Support installing multiple package versions :: CLPM supports installing
multiple versions of the same package simultaneously. This is an enabling
feature for managing project specific dependencies as well as global
dependencies.
+ Enable use of implementation specific features :: Because most of the work
is in a separate process, the main CLPM executable can be built using a
different Common Lisp implementation than the client that uses CLPM. This
means that CLPM can support only a small subset of implementations (and
use their specific features) without placing adverse constraints on
users.
;;;; CLPM Build Helper
(require :asdf)
(defpackage #:clpm-build
(:use #:cl)
(:import-from #:uiop
#:*stderr*
#:*stdout*
#:directory-exists-p
#:ensure-directory-pathname
#:find-symbol*
#:getenv
#:native-namestring
#:pathname-directory-pathname
#:run-program
#:with-current-directory
#:with-safe-io-syntax)
(:export #:build))
(in-package #:clpm-build)
(defvar *setup-file-pathname* *load-truename*
"The pathname to this file.")
(defvar *root-pathname* (pathname-directory-pathname *setup-file-pathname*)
"The pathname to the root directory of the CLPM release being built.")
(defvar *build-root-pathname* (merge-pathnames "build/"
*root-pathname*)
"The pathname to the root of the build directory. Defaults to build/ inside
*ROOT-PATHNAME*")
(defun run-commands-in-child-lisp (args &key env (lisp "sbcl"))
(loop
:for (name . val) :in env
:do (setf (getenv name) val))
(flet ((vomit (stream)
(with-safe-io-syntax ()
(loop
:for arg :in args
:do
(print arg stream)
(print arg *stdout*)))))
(run-program `(,lisp "--no-userinit" "--no-sysinit" "--disable-debugger")
:output :interactive
:error-output :interactive
:input #'vomit)))
(defun git-exists-p ()
(ignore-errors
(run-program '("git" "--version"))
t))
(defun git-version ()
(when (and (directory-exists-p (merge-pathnames ".git/"
*root-pathname*))
(git-exists-p))
(with-current-directory (*root-pathname*)
(let ((commit (run-program '("git" "rev-parse" "--short" "HEAD")
:output '(:string :stripped t)))
(dirty-p (not (zerop (nth-value 2
(run-program '("git" "diff-index" "--quiet" "HEAD")
:ignore-error-status t))))))
(format nil "~A~:[-dirty~;~]" commit dirty-p)))))
(defun build-clpm ()
(let ((clpm-pathname
#-:os-windows
(merge-pathnames "bin/clpm" *build-root-pathname*)
#+:os-windows
(merge-pathnames "bin/clpm.exe" *build-root-pathname*))
(clpm-client-pathname (merge-pathnames "bin/clpm-client.lisp" *build-root-pathname*))
(clpm-deps-root (merge-pathnames "src/clpm-deps/" *root-pathname*))
(cache-pathname (merge-pathnames "cl-cache/" *build-root-pathname*)))
(format *stderr* "Building `clpm` executable at ~A~%" clpm-pathname)
(run-commands-in-child-lisp
`((require :asdf)
(require :sb-posix)
(require :sb-bsd-sockets)
(format t "CL_SOURCE_REGISTRY: ~S~%" (getenv "CL_SOURCE_REGISTRY"))
(format t "ASDF_OUTPUT_TRANSLATIONS: ~S~%" (getenv "ASDF_OUTPUT_TRANSLATIONS"))
(when (uiop:featurep :os-windows)
(pushnew :drakma-no-ssl *features*))
(declaim (optimize debug))
(asdf:register-immutable-system :sb-posix)
(asdf:register-immutable-system :sb-bsd-sockets)
(asdf:load-system :clpm)
(setf (symbol-value (find-symbol* :*git-version* :clpm/version))
,(or (git-version)
""))
;; (defmethod asdf:output-files ((,(gensym) asdf:monolithic-concatenate-source-op)
;; (,(gensym) (eql (asdf:find-system :clpm-client))))
;; (values (list ,clpm-client-pathname) t))
(defmethod asdf:output-files ((,(gensym) asdf:program-op) (,(gensym) (eql (asdf:find-system :clpm))))
(values (list ,clpm-pathname) t))
(when (probe-file ,clpm-pathname)
(delete-file ,clpm-pathname))
;; (asdf:operate 'asdf:monolithic-concatenate-source-op :clpm-client)
;; (setf (symbol-value (uiop:find-symbol* :*client-contents* :clpm/client))
;; (uiop:read-file-string ,clpm-client-pathname))
(setf (symbol-value (uiop:find-symbol* :*deps-source* :clpm/deps))
(uiop:read-file-string ,(merge-pathnames "main.lisp" clpm-deps-root)))
(setf (symbol-value (uiop:find-symbol* :*deps-system* :clpm/deps))
(uiop:read-file-string ,(merge-pathnames "clpm-deps.asd" clpm-deps-root)))
(setf (symbol-value (uiop:find-symbol* :*deps-version* :clpm/deps))
(uiop:read-file-form ,(merge-pathnames "version.sexp" clpm-deps-root)))
;; (uiop:register-image-restore-hook
;; (lambda ()
;; (eval `(trace ,(uiop:find-symbol* :filter-requirements :clpm/resolve))))
;; nil)
(asdf:operate 'asdf:program-op :clpm))
:env `(("CL_SOURCE_REGISTRY"
. ,(format nil "~A/"
(namestring *root-pathname*)))
("ASDF_OUTPUT_TRANSLATIONS"
. ,(format nil "(:output-translations :ignore-inherited-configuration (:root (\"~A\" :implementation :**/ :*.*.*)))"
(namestring cache-pathname)))))))
(defun build (&key
(root-pathname *root-pathname*)
(build-root-pathname *build-root-pathname*))
(let ((*root-pathname* (ensure-directory-pathname root-pathname))
(*build-root-pathname* (ensure-directory-pathname build-root-pathname)))
(format *stderr*
"I will build CLPM from sources located at ~A~%The build file will be located at ~A~%~%~%"
*root-pathname*
*build-root-pathname*)
(build-clpm)))
;;;; CLPM-Licenses System Definition
(defsystem #:clpm-licenses
:description "System containing the licenses for CLPM and its dependencies."
:license "BSD-2-Clause"
:serial t
:components ((:module "licenses"
:components ((:static-file "alexandria")
(:static-file "anaphora")
(:static-file "archive")
(:static-file "asdf")
(:static-file "babel")
(:static-file "bordeaux-threads")
(:static-file "cffi")
(:static-file "chipz")
(:static-file "chunga")
(:static-file "cl-base64")
(:static-file "cl-fad")
(:static-file "cl-plus-ssl")
(:static-file "cl-ppcre")
(:static-file "cl-toml")
(:static-file "clon")
(:static-file "drakma")
(:static-file "esrap")
(:static-file "flexi-streams")
(:static-file "iterate")
(:static-file "local-time")
(:static-file "log4cl")
(:static-file "log4cl.NOTICE")
(:static-file "named-readtables")
(:static-file "osicat")
(:static-file "puri")
(:static-file "sbcl")
(:static-file "split-sequence")
(:static-file "trivial-features")
(:static-file "trivial-garbage")
(:static-file "trivial-gray-streams")
(:static-file "trivial-types")
(:static-file "usocket")))
(:file "src/clpm-licenses/main")))
;;;; CLPM System Definition
#-:asdf3.1
(error "Requires ASDF >=3.1")
(defsystem #:clpm
:version (:read-file-form "src/clpm/version.lisp" :at (2 2))
:description "A Common Lisp package Manager"
:license "BSD-2-Clause"
:pathname "src/clpm/"
:class :package-inferred-system
:entry-point "clpm/cli/entry:main"
:depends-on (#:clpm/clpm))
#+TITLE: CLPM Bundle
#+AUTHOR: Eric Timmons
#+EMAIL: etimmons@mit.edu
This file describes the "bundle" capabilities of CLPM. Bundles are meant to ease
the problems of repeatable deployment and testing in the CL world - i.e.,
helping handle where the rubber meets the road. While ASDF famously does not
allow developers to specify upper bounds on dependency versions (only lower
bounds) [fn:1] [fn:2], CLPM's bundle capability allows exactly that, in addition
to specifying that some dependency should be satisfied by some unreleased
version directly from source control (great for developing projects in multiple
repos simultaneously!).
A CLPM bundle is a collection of systems locked at specific versions. A bundle
is fully defined by a =clpmfile.lock= file, which is itself created from a
=clpmfile= file. Once a lock file is generated, the bundle commands allow you to
perform tasks such as packaging the whole bundle for easy deployment/sharing and
running Lisp processes that can load systems in the bundle with zero
configuration or helpers needed (other than ASDF, of course).
* Files
** clpmfile
A =clpmfile= describes what projects or systems a user wants to place in a
bundle and any constraints she has on the versions of those objects. It is a
file that consists of a series of directives where each directive is a list
starting with a keyword. The directives are described below. Additionally, the
first directive in the file must be the ~:api-version~ directive. This
document describes version ="0.1"=.
*** ~:api-version~
This directive must be the first to appear in the file. It consists of one
argument that must be a string naming the clpmfile version the rest of the
file is described in. Example:
#+begin_src common-lisp
(:api-version "0.1")
#+end_src
*** ~:source~
Each clpmfile must include at least one source directive. This tells CLPM
which sources it is allowed to search for dependencies in. The first
argument to this directive is the name of the source as a string. The
remainder is a plist describing the source. For example, to use the standard
quicklisp distribution, you can write:
#+begin_src common-lisp
(:source "quicklisp"
:url "https://beta.quicklisp.org/dist/quicklisp.txt"
:force-https t)
#+end_src
All source directives must come immediately after the api-version directive.
*** ~:gitlab~
The gitlab directive specifies a requirement that must be satisfied by a git
repository hosted on a gitlab server. Unless specified otherwise, every
explicitly defined system (and their dependencies) are included in the
bundle.
#+begin_src common-lisp
(name &key (host "gitlab.com") repo branch commit tag systems)
#+end_src
+ ~name~ :: must be a string, unique in the file.
+ ~:host~ :: is the hostname of the gitlab server. Defaults to gitlab.com.
+ ~:repo~ :: is a string naming the path to the repository.
+ ~:branch~ :: If specified, must be a string naming the branch to use. If
none of branch, commit, or tag are specified, the branch
defaults to the default branch from the server.
+ ~:commit~ :: If specified, must be a string naming the commit to use.
+ ~:tag~ :: If specified, must be a string naming the tag to use.
+ ~:systems~ :: A list naming the systems to require in the bundle. If NIL,
all explicitly defined systems are required.
*** ~:asd~
This directive states that the specified asd file should be included in the
bundle. The arguments it accepts are:
#+begin_src common-lisp
(asd-path &key systems)
#+end_src
+ ~asd-path~ :: Must be a string providing the path to the asd file,
relative to the clpmfile.
+ ~systems~ :: What systems defined by the ASD file should be included. Must
be a list of strings. If NIL, then all systems explicitly
defined in the asd file are required and have their
dependencies satisfied.[fn:3]
** clpmfile.lock
The lock file defines all releases included in the bundle. This consists of
every top level requirement specified in the clpmfile and their
dependencies. When the dependencies are resolved, any system fetched from
source control is assumed to satisfy any requirements placed on it. No formal
commitment to syntax for this file is guaranteed yet.
* Commands
** =clpm bundle install=
If the lock file exists, load it and ensure that all releases included in the
bundle are installed locally. If the lock file does not exist, create it and
then ensure all releases are installed.
** =clpm bundle exec=
Execute the specified command (following =exec=) in an environment where ASDF
will have access to all systems included in the bundle with no extra
configuration. Requires the lock file to exist.
* Comparisons
In the Common Lisp world, CLPM's bundle is most similar to [[https://github.com/fukamachi/qlot][Qlot]]. Unlike Qlot,
however, CLPM has the following features:
+ Has no dependency on roswell.
+ Is distributed both as source and an executable.
+ Works harder to determine dependencies of systems fetched from git.
+ Allows for more complex version specifications in the clpmfile.
+ Allows for easily cloning remote repositories over HTTPS with
authentication.
Like Qlot, this portion of CLPM was heavily influenced by Ruby's bundler.
* Footnotes
[fn:1] https://bugs.launchpad.net/asdf/+bug/1183179
[fn:2] I personally didn't like this restriction at first, but upon much
reflection realized that Faré's argument made a lot of sense. That realization
was one of the impetuses to designing the bundle capabilities of CLPM.
[fn:3] Only the top level system of a package-inferred-system is considered to
be explicitly defined. If you require dependencies of subsystems that the top
level system does not depend on to be satisfied, you must list those systems
explicitly.
#+TITLE: CLPM Configuration
CLPM is configured using toml configuration files.
* Configuration Values
** git
*** remotes
This table contains options for connecting to remote git servers. Each key
in the table must be the hostname of the remote server (in all
lowercase). The value of each key is a table containing the following
values:
| key | Type | value |
|----------+--------+----------------------------------------------------------------------------------|
| type | String | The type of the remote server |
| method | String | The preferred method for fetching code. Can be one of "https" (default) or "ssh" |
| username | String | The username to use when fetching over https. |
| password | String | The password to use when fetching over https. |
** http
*** headers
This table provides a mechanism to specify headers to send to servers when
downloading files over HTTP(S). This does *not* work when fetching git repos
over HTTP(S), see [[*remotes]] instead.
Each key in this table is the hostname of a server (in all lowercase) and
the values are tables, where each table maps a header name to either a
string or another table. If the header name is mapped to a string, that
string is sent as the value of the header. If the header name is mapped to
another table, the following keys are recognized. Behavior is undefined if
both =value= and =exec= are provided.
| key | type | value |
|-------------+---------+------------------------------------------------------------------------------|
| secure-only | Boolean | If true, this header is sent only over HTTPS connections. Defaults to false. |
| value | String | This value is used as the value of the header. |
| exec | String | Must point to an executable that prints out the value to use to its stdout. |
For example, you can use the following to access release tarballs from
private projects on gitlab.com using a Personal Access Token.
#+begin_src conf-toml
[http.headers."gitlab.com"]
private-token = { exec = "command-that-prints-my-pat" , secure-only = true}
#+end_src
** grovel
*** sandbox
This table lets you configure the sandboxing mechanism used when locally
determining (groveling for) the dependencies of some ASDF system. Currently
accepts the following keys:
| key | type | value |
|--------+--------+-----------------------------------------------|
| method | string | The sandbox method to use. Defaults to "auto" |
#+TITLE: CLPM Sources
A repository of Common Lisp projects is called a "source". The source you are
most likely familiar with is Quicklisp, but CLPM is designed to use multiple
sources and source types. This allows you to easily use CLPM for things such as
internal projects that can't be release to private repositories.
Currently, the only type of source that CLPM publicly supports is a Quicklisp
style source. The author is currently working on and testing a new source type
that improves upon Quicklisp, though (such as tracking of individual system
versions). If you are interested in trying out that source type, feel free to
reach out.
The type of every source can be specified. If not specified, CLPM will attempt
to guess the source type based on the file located at the specified URL. It is
recommended that the type be specified, however.
* Quicklisp
A Quicklisp style source contains a subscription metadata file (such as the
primary Quicklisp one located at
https://beta.quicklisp.org/dist/quicklisp.txt). Additionally, there must a
file whose name is the same but suffixed with "-versions" (e.g.,
https://beta.quicklisp.org/dist/quicklisp-versions.txt) that points to the
dist file for every version of the distribution.
The type designator for a Quicklisp like source is "quicklisp". A quicklisp
distribution can be configured with the following options:
+ =url= :: The URL to the subscription metadata file.
+ =force-https= :: If true, the protocol for all URLs is changed to https
before downloading any files. This is useful because all
URLs contained in the primary Quicklisp distribution have
http as the protocol even though they are also served over
https.
The following table can be added to your CLPM config file to use the primary
Quicklisp distribution.
#+begin_src conf-toml
[sources.quicklisp]
type = "quicklisp"
url = "http://beta.quicklisp.org/dist/quicklisp.txt"
force-https = true
#+end_src
This directory contains the licenses for every dependency of CLPM that is
included in it when built. When possible, the license files are just symlinks to
the license files provided by that dependency's upstream. When that is not
possible, the file is a plain text file containing the license as extracted from
one of its source files and/or system definition file.
During the build process, these licenses are included in the generated image and
can be printed with `clpm license-info`.
../ext/alexandria/LICENCE
\ No newline at end of file
../ext/anaphora/LICENSE
\ No newline at end of file
../ext/archive/LICENSE
\ No newline at end of file
Copyright (c) 2001-2018 Daniel Barlow and contributors
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
../ext/babel/COPYRIGHT
\ No newline at end of file
../ext/bordeaux-threads/LICENSE
\ No newline at end of file
../ext/cffi/COPYRIGHT
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment