Commit 01fa0fc2 authored by Eric Timmons's avatar Eric Timmons

First commit

parents
Copyright 2020 Eric Timmons
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#+TITLE: Common Lisp Project Index
#+AUTHOR: clpm-devel@common-lisp.net
This project provides the specification for the Common Lisp Project Index
(CLPI) as well as a reference implementation of a library to interact with and
create indices. A project index stores metadata about projects and systems
defined by those projects, with a primary use case of allowing package managers
to know which releases exist and to trace the dependencies of systems.
In an ideal world, much of the system information could be stored in the index
by merely supplying the .asd files that define the systems. However, .asd files
can include arbitrary code, require that other systems need to be loaded in
order to understand the system definition, and refer to other files (such as
for version numbers, system descriptions, or even children system definitions
in the case of package-inferred-systems). Therefore, the index exists as a
purely declarative specification of projects and their systems.
* Versions
The CLPI specification is constantly evolving. As such, the specification is
versioned and indices must declare which version(s) of the specification they
support. Versions 0.1 and 0.3 have existed as undocumented implementations
within the CLPM (Common Lisp Package Manager) code. The first version
documented and supported by this project is v0.4. The specification is
located at [[file:specs/clpi-0.4.org][specs/clpi-0.4.org]].
* Taxonomy
This section attempts to ground the definitions of several terms used
throughout this documentation.
** System
A system refers to a system defined using ASDF. Systems have a name, a
version, dependencies, a license, and a description.
** Project
A project is a cohesive unit of one or more systems that are developed
together. Projects have a name, maintainers, authors, source repository,
homepage, and system(s).
** Release
A release is a versioned snapshot of a project.
** System release
A system release is a versioned snapshot of a system, contained within a
release.
;;;; clpi-test System Definition
#-:asdf3.2
(error "Requires ASDF >=3.2")
;; Not necessary, but easier to have when using SLIME.
(in-package :asdf-user)
(defsystem #:clpi-test
;;:version (:read-file-form "src/version.lisp" :at (2 2))
:description "Test system for clpi system."
:license "BSD-2-Clause"
:pathname "test/"
:class :package-inferred-system
:depends-on (#:clpi-test/clpi-test))
;;;; CLPI System Definition
;;;;
;;;; This software is part of CLPI. See README.org for more information. See
;;;; LICENSE for license information.
#-:asdf3.2
(error "Requires ASDF >=3.2")
;; Not necessary, but easier to have when using SLIME.
(in-package :asdf-user)
(defsystem #:clpi
:version "0.0.1"
:description "A library to interact with and create Common Lisp Project Indices."
:license "BSD-2-Clause"
:pathname "src/"
:class :package-inferred-system
:depends-on (#:clpi/clpi)
:in-order-to ((test-op (load-op "clpi-test")))
:perform (test-op (op c) (unless (symbol-call :fiveam :run! :clpi)
(error "Test failure"))))
#+TITLE: CLPI Specification v0.4
#+AUTHOR: clpm-devel@common-lisp.net
This document provides the specification of version 0.4 of the Common Lisp
Project Index.
An index consists of a series of objects that contain information about the
projects and systems defined in the index. Two common examples of indices would
be an index located on a file system (where each object is a seperate file on
the file system) and an index accessed over HTTP (where each object is accessed
via a separate URL).
Each object must contain s-expressions that can be =READ= by a Common Lisp
implementation. All symbols in the s-expressions must be keywords.
Any plist may contain keys beyond those defined by this specification. However,
as there is no central registry of extensions to the spec, all such keywords
*must* be prefixed with an identifier folowed by a =/=. For example, the
=ql-clpi= project prefixes all custom keys (such as version aliases) with
=ql/=. New keys for any plist may be added to this version of the specification
so long as the changes are backwards compatible.
As many users of an index may desire to work offline and/or not need access to
the entire index, it is designed to be somewhat efficient when syncing across a
network. In this case the canonical version of the complete index is called the
primary and the local (potentially incomplete) version of the index is called
the secondary.
* Index metadata
The object =metadata= must exist and contain a single plist. The defined keys
are:
+ =:clpi-version= :: Required. A string naming the version of the CLPI
specification used by this index. Must be ="0.4"=.
+ =:canonical-url= :: Optional. A string containing the canonical URL to this
index's metadata. Used when hosting a primary index for remote access.
* Index
The object =index= must exist. This object is a lightweight summary of all
projects, systems, and releases contained within the index. It contains a
single plist with the keys:
+ =:projects= :: A list of project information, described in [[*Project index]].
+ =:systems= :: A list of system information, described in [[*System index]].
As this object is typically fetched from the primary by the secondaries on a
regular basis, other keys (including in sub plists) are allowed, but
discouraged in order to keep the size to a minimum.
** Project index
This list contains all projects defined within the index as well as each
project's releases. Each element in the project index must be of the form
=(project-name . project-plist)= where =project-name= is a string naming the
project and =project-plist= is a plist containing the following keys:
+ =:releases= :: A list of strings providing the versions of each release of
the project.
** System index
This list contains the names of all systems defined within the index as well
as which project releases provide them. Every element in the system index
must be of the form =(system-name . system-plist)=, where =system-name= is a
string containing the name of the system and =system-plist= is a list
containing the following keys:
= =:releases= :: A list of releases of the system. Each element is a list
with one or two members. The first member is a [[*Project release description]].
The second element is the ASD version of the system (if one is defined). If
the second element is missing, it is assumed the ASD version of the system
is =NIL=.
* Project metadata
For every project in the index, the object =projects/PROJECT-NAME/metadata=
must exist, where =PROJECT-NAME= is the URL-encoded project name.
This object must contain a single plist. The defined keys are:
+ =:repo= :: A VCS repository as described in [[*Repository Description]].
+ =:authors= :: A list of person descriptions as described in [[*Person
Description]].
+ =:maintainers= :: A list of person descriptions as described in [[*Person
Description]].
+ =:homepage= :: A string containing the URL of the project's homepage.
+ =:epoch= :: An integer or NIL/missing whose purpose is further described in
[[*Project releases]].
+ =:version-scheme= :: A keyword describing how this project versions its
releases. =:date= implies a date based version that is comparable with
=#'string<=. =:semantic= implies version numbers that are compatible with
[[https://semver.org/][Semver v2]].
* Project releases
For every project in the index, the object =project/PROJECT-NAME/releases= or
=projects/PROJECT-NAME/releases-EPOCH= must exist, where =PROJECT-NAME= is
the URL-encoded project name and =EPOCH= is the epoch defined in the project
metadata. If the epoch is NIL/not provided, only the former must exist,
otherwise the latter must exist.
This object contains information about each release of the project. Every
sexp must be of the form: =(version . release-plist)=, where =version= is a
string containing the release's version and =release-plist= is a plist with
the following keys:
+ =:url= :: Required. A string containing the URL of an archive for this
release. This archive must contain a single top level folder.
+ =:archive-type= :: A keyword describing the type of archive. Defined
keywords are: =:tar.gz= and =:zip=. Defaults to =:tar.gz= if not specified.
+ =:size= :: The size of the release archive in bytes.
+ =:md5= :: The md5sum of the release archive.
+ =:systems= :: An alist mapping paths to system files within the release
(relative to the archive's top level folder) to lists of [[*System description]].
** System description
A system description is an alist mapping a string system name to a plist
with information about that system. The defined keys in the plist are:
+ =:version= :: A string containing the version of the system. If non-NIL,
must be a dotted sequence of numbers (this requirement comes from ASDF,
unfortunately).
+ =:description= :: A string containing a description of the system.
+ =:license= :: The license of the system. Can be an arbitrary sexp.
+ =:dependencies= :: A list of dependencies for this system. For a
description of the allowed forms of dependencies, see [[*System dependency]].
If an integer epoch is provided, this object must only be appended to. This
allows secondary indices to more efficiently transfer only the new
releases. If for any reason this object needs to be modified (other than
appending), the project's epoch must be incremented and a new object used
instead.
If no epoch (or a NIL epoch) is provided, this object may change in any way.
* System metadata
For every primary system in the index, the object
=systems/SYSTEM-NAME/metadata= must exist, where =SYSTEM-NAME= is the
URL-encoded primary system name.
This object must contain a single plist. The defined keys are:
+ =:primary-project= :: A string naming the current home project of this
system.
* System releases
For every primary system in the index, the object
=systems/SYSTEM-NAME/releases= or =systems/SYSTEM-NAME/releases-EPOCH= must
exist, where =SYSTEM-NAME= is the URL-encoded primary system name and =EPOCH=
is the epoch defined in the system metadata. If the epoch is NIL/not
provided, only the former must exist, otherwise the latter must exist.
This object contains information about each release of the system. Every sexp
must be of the form: =(release . release-info)=, where =release= is a
[[*Project release description]] and =release-info= is a plist with the following
keys:
+ =:system-file= :: Required. A string containing the path to the ASD file
that defines this system, relative to the top level folder in the release's
archive.
+ ::
+ =:version= :: NIL or a string containing the version of the system. If
non-NIL, must be a dotted sequence of numbers (this requirement comes from
ASDF, unfortunately).
+ =:description= :: A string containing a description of the system.
+ =:license= :: The license of this system. Can be an arbitrary sexp.
+ =:dependencies= :: A list of dependencies for this system. See [[*System
dependency]] for a description of the allowed forms of dependencies.
+ =:secondary-systems= :: A list of secondary systems with the same primary
name. See [[*Secondary system release]] for a description of secondary systems.
If an integer epoch is provided, this object must only be appended to. This
allows secondary indices to more efficiently transfer only the new
releases. If for any reason this object needs to be modified (other than
appending), the system's epoch must be incremented and a new object used
instead.
If no epoch (or a NIL epoch) is provided, this object may change in any way.
** Secondary system release
A secondary system is a system whose name contains a =/=
character. Secondary systems may be explicitly defined by a =defsystem=
form, or implicitly defined (as is the case when using ASDF's
package-inferred-systems). Due to the increasing prevalence of
package-inferred-systems, and subsequent increase in number of systems that
a single release can have, v0.4 of the specification does not create an
object for each individual system; instead, the data for secondary systems
is in the same object as the data on the primary system (similar to how .asd
files work).
Each secondary system release is described by a sexp of the form =(name
. info)= where =name= is a string containing the name of the system and
=info= is a plist with the following keys:
+ =:version= :: NIL or a string containing the version of the system. If
non-NIL, must be a dotted sequence of numbers (this requirement comes from
ASDF, unfortunately).
+ =:description= :: A string containing a description of the system.
+ =:license= :: The license of this system. Can be an arbitrary sexp.
+ =:dependencies= :: A list of dependencies for this system. See [[*System
dependency]] for a description of the allowed forms of dependencies.
If =:version= or =:license= is missing from a secondary system, it is
assumed to be the same as the primary system's.
* System dependency
A system dependency can take one of four forms:
1. A string naming a system.
2. A list with =:require= as the first element and a string naming a module
as the second element.
3. A list of the form =(:version SYSTEM-NAME VERSION-SPECIFIER)=, where
=SYSTEM-NAME= is a string naming a system and =VERSION-SPECIFIER= is an
ASDF version string (dot separated numbers only). This states the minimum
version of the system that is needed.
4. A list of the form =(:feature FEATURE-EXPRESSION SYSTEM-DEPENDENCY)= where
=FEATURE-EXPRESSION= denotes either a (keyword!) feature, negation,
disjunction, or conjunction of feature expressions (see the ASDF manual)
and =SYSTEM-DEPENDENCY= is a dependency as defined by this section.
* Person Description
This is used to describe a person (or other entity). It is either a string
containing the name of the entity or a list of two elements. The first
element must be a string naming the entity and the second must be a string
containing an email address.
* Project release description
A project release description is a list of two elements. The first element is
a string naming a project. The second element is a string naming the version
of a release of that project.
* Respository Description
This is used to describe a source control repository. It must be a list of
the form =(REPO-TYPE . REPO-ARGS)= where =REPO-TYPE= is a keyword and
=REPO-ARGS= is a plist. This following repo types are defined:
** Gitlab Repository
Describes a git repository hosted on a Gitlab server. The =REPO-TYPE= must
be =:gitlab=. The following args are defined:
+ =:host= :: A string containing the hostname of the gitlab server. Defaults
to ="gitlab.com"= if not provided.
+ =:path= :: Required. A string containing the path to the repository. Must
not end in =.git=.
** Github Repository
Describes a git repository hosted on a Github server. The =REPO-TYPE= must
be =:github=. The following args are defined:
+ =:host= :: A string containing the hostname of the gitlab server. Defaults
to ="github.com"= if not provided.
+ =:path= :: Required. A string containing the path to the repository. Must
not end in =.git=.
;;;; CLPI - Common Lisp Project Index
;;;;
;;;; This software is part of CLPI. See README.org for more information. See
;;;; LICENSE for license information.
(uiop:define-package #:clpi/clpi
(:nicknames #:clpi)
(:use #:cl
#:clpi/defs
#:clpi/dual-index
#:clpi/file-index
#:clpi/http-index
#:clpi/metadata
#:clpi/project
#:clpi/release
#:clpi/system
#:clpi/system-file
#:clpi/system-release)
(:reexport #:clpi/defs
#:clpi/dual-index
#:clpi/file-index
#:clpi/http-index
#:clpi/project
#:clpi/release
#:clpi/system
#:clpi/system-file
#:clpi/system-release)
(:export #:metadata-value))
(in-package #:clpi/clpi)
;;;; Defs - Consolidates all defs into one package
;;;;
;;;; This software is part of CLPI. See README.org for more information. See
;;;; LICENSE for license information.
(uiop:define-package #:clpi/defs
(:use #:cl
#:clpi/index-defs
#:clpi/project-defs
#:clpi/release-defs
#:clpi/system-defs
#:clpi/system-file-defs
#:clpi/system-release-defs)
(:reexport #:clpi/index-defs
#:clpi/project-defs
#:clpi/release-defs
#:clpi/system-defs
#:clpi/system-file-defs
#:clpi/system-release-defs))
(in-package #:clpi/defs)
;;;; Dual Index Definitions
;;;;
;;;; This software is part of CLPI. See README.org for more information. See
;;;; LICENSE for license information.
(uiop:define-package #:clpi/dual-index
(:use #:cl
#:alexandria
#:clpi/index-defs
#:clpi/index
#:clpi/project-defs)
(:export #:dual-index))
(in-package #:clpi/dual-index)
(defclass dual-index (index)
((primary
:initarg :primary
:reader dual-index-primary)
(secondary
:initarg :secondary
:reader dual-index-secondary)
(updated-paths
:initform (make-hash-table :test 'equal)
:reader dual-index-updated-paths))
(:documentation
"A logical view of an index that is backed by two separate indices, the
primary and the secondary. All reads prefer the secondary, but if some data is
missing, it will be transparently read from the primary and placed into the
secondary. Additionally, any metadata file and the project/system indices will
be read from the primary on first access if the local copies are older than 10
minutes old."))
(defmethod index-backing-objects-exist-p ((index dual-index))
t)
(defmethod index-release-data-missing ((index dual-index) project release)
(let* ((path (project-releases-path project))
(primary (dual-index-primary index))
(secondary (dual-index-secondary index))
(secondary-size (index-object-size secondary path)))
(with-open-index-object-stream (primary-stream primary path
:binary-p t
:offset secondary-size)
(index-object-append secondary path primary-stream))))
(defun path-should-be-fetched-if-old-p (index path)
(and (or (equal path "project-index")
(equal path "system-index")
(ends-with-subseq "metadata" path))
(not (gethash path (dual-index-updated-paths index)))))
(defmethod open-index-object-stream ((index dual-index) path
&key signal-if-older-than
binary-p offset)
(let ((primary (dual-index-primary index))
(secondary (dual-index-secondary index)))
(flet ((read-from-primary ()
(with-open-index-object-stream (s primary path)
(index-object-write secondary path s)
(setf (gethash path (dual-index-updated-paths index)) t))))
(handler-bind ((index-object-missing
(lambda (c)
(read-from-primary)
(continue c)))
(index-object-out-of-date
(lambda (c)
(declare (ignore c))
(when (path-should-be-fetched-if-old-p index path)
(read-from-primary)))))
(open-index-object-stream secondary path
:signal-if-older-than (or signal-if-older-than
(- (get-universal-time)
(* 10 60)))
:binary-p binary-p
:offset offset)))))
;;;; File based index
;;;;
;;;; This software is part of CLPI. See README.org for more information. See
;;;; LICENSE for license information.
(uiop:define-package #:clpi/file-index
(:use #:cl
#:clpi/defs
#:clpi/index
#:clpi/metadata
#:clpi/utils)
(:export #:file-index))
(in-package #:clpi/file-index)
(defclass file-index (index)
((root
:initarg :root
:reader file-index-root
:documentation
"A directory pathname to the root of the index."))
(:documentation
"A Common Lisp Project Index backed by storage on a file system."))
(defmethod initialize-instance :before ((index file-index) &key root &allow-other-keys)
(unless (uiop:directory-pathname-p root)
(error ":ROOT must be a directory pathname")))
(defmethod index-backing-objects-exist-p ((index file-index))
(probe-file (metadata-path index)))
(defmethod index-object-append ((index file-index) path in-stream)
(let ((pn (merge-pathnames path (file-index-root index))))
(ensure-directories-exist pn)
(with-open-file (s pn
:element-type (stream-element-type in-stream)
:direction :output
:if-exists :append
:if-does-not-exist :create)
(uiop:copy-stream-to-stream in-stream s
:element-type (stream-element-type in-stream)))))
(defmethod index-object-size ((index file-index) path)
(let ((pn (merge-pathnames path (file-index-root index))))
(with-open-file (s pn
:element-type '(unsigned-byte 8)
:if-does-not-exist nil)
(when s
(file-length s)))))
(defmethod index-object-write ((index file-index) path in-stream)
(let ((pn (merge-pathnames path (file-index-root index))))
(ensure-directories-exist pn)
(with-open-file (s pn
:element-type (stream-element-type in-stream)
:direction :output
:if-exists :supersede)
(uiop:copy-stream-to-stream in-stream s))))
(defmethod index-save ((index file-index))
(when (metadata-dirty-p (metadata index))
(let ((pn (merge-pathnames "metadata" (file-index-root index))))
(ensure-directories-exist pn)
(with-open-file (s pn
:direction :output
:if-exists :supersede)
(metadata-write-to-stream index s))))
(let ((pn (merge-pathnames "index" (file-index-root index))))
(ensure-directories-exist pn)
(with-open-file (s pn
:direction :output
:if-exists :supersede)
(let ((plist (index-plist index)))
(with-clpi-io-syntax ()
(pprint plist s)))))
(dolist (p (index-projects index))
(let ((dirty-p (project-dirty-p p))
(appendable-p (project-changes-appendable-p p))
(epoch (project-epoch p)))
(when dirty-p
(when (and epoch (not appendable-p))
(incf (project-epoch p)))
(let ((pn (merge-pathnames (project-releases-path p) (file-index-root index))))
(ensure-directories-exist pn)
(with-open-file (s pn
:direction :output
:if-exists (if appendable-p :append :supersede)
:if-does-not-exist :create)
(if (and appendable-p epoch)
(project-write-changes-to-stream p s)
(project-write-to-stream p s)))))
(when (metadata-dirty-p (metadata p))
(let ((pn (merge-pathnames (metadata-path p) (file-index-root index))))
(ensure-directories-exist pn)
(with-open-file (s pn
:direction :output
:if-exists :supersede)
(metadata-write-to-stream p s))))))
(dolist (system (index-systems index))
(when (metadata-dirty-p (metadata system))
(let ((pn (merge-pathnames (metadata-path system) (file-index-root index))))
(ensure-directories-exist pn)
(with-open-file (s pn
:direction :output
:if-exists :supersede)
(metadata-write-to-stream system s))))))
(defmethod open-index-object-stream ((index file-index) path
&key signal-if-older-than
binary-p offset)
(let ((pn (merge-pathnames path (file-index-root index)))
(open-args nil))
(when binary-p
(push '(unsinged-byte 8) open-args)
(push :element-type open-args))
(unless (probe-file pn)