C

clamber

Clamber is a standalone application for managing documents.

With time it will mature enough to be considered serious (hopefully), right now it is meant as an accompany material for ECL Quarterly issue "Creating Common Lisp project".

Name Last Update
img/vol5 Loading commit data...
.gitignore Loading commit data...
README.md Loading commit data...
clamber.asd Loading commit data...
clamber.lisp Loading commit data...
cli.lisp Loading commit data...
tests.lisp Loading commit data...

Introduction

A common question heard from the Common Lisp newcomers is:

How to create my own application with Common Lisp?

Numerous concepts like packages, Quicklisp, modules and ASDF bring the confusion, which is only deepened by a wide range of implementations and foreign to the new programmer developing paradigm of working on a live image in the memory.

This post is a humble attempt to provide a brief tutorial on creating a small application from scratch. Our goal is to build a tool to manage document collection. Due to the introductory nature of the tutorial we will name our application "Clamber".

We will start with a quick description of what should be installed on the programmer's system (assumed operating system is Linux). Later we will create a project boilerplate with the quickproject, define a protocol for the software, write the application prototype (ineffective naive implementation), provide the command line interface with Command Line Option Nuker. This is where the first part ends.

Second part will be published on McCLIM blog and will show how to create a graphical user interface for our application with McCLIM.

Afterwards (in a third part in next ECL Quarterly) we will take a look into some considerations on how to distribute the software in various scenarios:

  • Common Lisp developers perspective with Quicklisp,
  • ordinary users with system-wide package managers with ql-to-deb,
  • source code distribution to clients with Qucklisp Bundles,
  • binary distribution (closed source) with ADSF prebuilt-system,
  • as a shared library for non-CL applications with ECL.

Obviously a similar result may be achieved using different building blocks and all choices reflect my personal preference regarding the libraries I use.

How to distribute the software

Before we jump into the project creation and actual development I want to talk a little about the software distribution. We may divide our target audience in two groups – programmers and end usersprogrammers. Sometimes it is hard to tell the difference.

Programmers want to use our software as part of their own software as a dependency. This is a common approach in FOSS applications, where we want to focus on the problem we want to solve, not the building blocks which are freely available (what kind of freedom it is depends on the license). To make it easy to acquire such dependencies the Quicklisp project was born. It is a package manager.

End users aren't much concerned about the underlying technology. They want to use the application in the most convenient way to them. For instance average non-programming Linux user would expect to find the software with theirs system distribution package manager. Commercial client will be interested in source code with all dependencies with whom the application was tested.

Proposed solution is to use Quicklisp during the development and bundle the dependencies (also with Quicklisp) when the application is ready. After that operation our source code doesn't depend on the package manager and we have all the source code available, what simplifies further distribution.

What are Common Lisp systems?

Notion of "system" is unknown to the Common Lisp specification. It is a build-system specific concept. Most widely used build-system in 2016 is ASDF. System definition is meant to contain information essential for building the software – application name, author, license, components and dependencies. Unfortunately ADSF doesn't separate system definitions from the source code and asd format can't be considered declarative. In effect, we can't load all system definitions with certainty that unwanted side-effects will follow.

Development environment configuration

We will only outline steps which are necessary to configure the development environment. There are various tutorials on how to do that which are more descriptive.

  1. Install Emacs and SBCL1:

    These two packages should be available in your system package manager (if it has one).

  2. Install Quicklisp:

    Visit https://www.quicklisp.org/beta/ and follow the instructions. It contains steps to add Quicklisp to Lisp initialization file and to install and configure SLIME. Follow all these instructions.

  3. Start Emacs and run Slime:

    To run Slime issue M-x slime in Emacs window.

These steps are arbitrary. We propose Linux + SBCL + Emacs + Quicklisp + SLIME setup, but alternative configurations are possible.

How to create a project

Quickproject is an excellent solution for this task because it is very simple tool with a well defined goal – to simplify creating basic project structure.

The simplest way of creating a new one is loading the quickproject system with Quicklisp and calling the appropriate function. Issue the following in the REPL:

(ql:quickload 'quickproject)
(quickproject:make-project #P"~/quicklisp/local-projects/clamber/"
                           :depends-on '(#:alexandria)
                           :author "Daniel Kochmański <daniel@turtleware.eu>"
                           :license "Public Domain")

That's it. We have created a skeleton for our project. For now, we depend only on alexandria – public domain utility library. List of dependencies will grow during the development to reflect our needs. Go to the clamber directory and examine its contents.

Now we will customize the skeleton. I prefer to have one package per file, so I will squash package.lisp and clamber.lisp into one. Moreover, README.txt will be renamed to README.md, because we will use markdown format for it.

To avoid clobbering the tutorial with unnecessary code we put only interesting parts here. Complete steps are covered in the application GIT repository available here:

https://gitlab.common-lisp.net/dkochmanski/clamber

We propose to clone the repository and track the progress with the subsequent commits and this tutorial.

Writing the application

Here is our application Clamber informal specification:

  • Application will be used to maintain a book collection,
  • Each book has associated meta-information (disregarding the underlying book file format),
  • Books may be organized with tags and shelves,
  • Book may be only on one shelf, but may have multiple tags,
  • Both CLI and GUI interfaces are a required,
  • Displaying the books is not part of the requirements (we may use external programs for that).

Protocol

First we will focus on defining a protocol. Protocol is a functional interface to our application. We declare how external modules should interact with it. Thanks to this approach we are not tied to the implementation details (exposing internals like hash tables or class slot names would hinder us during the future refactoring, or could cause changes which are not backward compatible).

;;; Clamber book management protocol

;;; Requirements explicitly list that books has to be organized by
;;; shelves and tags. Book designator is used to identify books (it
;;; has to be unique). Protocol doesn't mandate designator type. It
;;; may be a unique name, pathname, URL or any arbitrary
;;; object. Other args (in form of keys) are meant to contain
;;; meta-information.
(defgeneric insert-book (book-designator &rest args
                         &key shelf tags &allow-other-keys)
  (:documentation "Creates a book entity associated to a given ~
   `shelf' and `tags'."))

;;; We need to bo able to remove book. We need only the designator for
;;; that.
(defgeneric delete-book (book-designator)
  (:documentation "Removes a book entity from the system."))

;;; We may search for books according to various
;;; criteria. `book-designator' is definite. It is possible to extend
;;; the find functionality to support other criteria. Book must match
;;; *all* supplied criteria.
(defgeneric find-books (&rest args
                        &key book-designator shelf tags &allow-other-keys)
  (:documentation "Returns a `sequence' of books matching the ~
  requirements."))

;;; We access books by their designators, but `find-books' returns a
;;; list of opaque objects. This function is needed for coercion from
;;; these objects to the designators. Sample usage:
;;; 
;;; (map 'list #'book-designator (find-books :shelf "criminal"))
(defgeneric book-designator (book)
  (:documentation "Extract book designator from opaque `book' object."))

This code is put in clamber.lisp file. It is important to remember, that :documentation clause in defgeneric is meant only for programmers who use our library (to provide a short reminder of what the function does) and shouldn't be considered being the final documentation. Especially docstrings are not documentation.

Comments are meant for programmers who work on our library (extend the library or just read the code for amusement). Their meaning is strictly limited to the implementation details which are irrelevant for people who use the software. Keep in mind, that comments are not reference manual.

Implementation prototype

Our initial implementation will be naive so we can move forward faster. Later we could rewrite it to use a database. During the prototyping programmer may focus on the needed functionality and modify the protocol if needed.

This is a tight loop of gaining the intuition and adjusting rough edges of the protocol. At this phase you mustn't get too attached to the code so you can throw it away without hesitation. More time you spend on coding more attached to the code you are.

;;; Implementation

;;; At start we are going to work on in-memory database.
(defparameter *all-books* (make-hash-table) "All defined books.")

;;; Note, that when we define `:reader' for the slot `designator' we
;;; actually implement part of the protocol.
(defclass book ()
  ((designator :type symbol   :initarg :des :reader book-designator)
   (shelf      :type string   :initarg :shl :reader book-shelf)
   (tags       :type sequence :initarg :tgs :reader book-tags)
   (meta :initarg :meta :accessor book-information)))

;;; `title' and `author' are enlisted for completion.
(defmethod insert-book ((designator symbol) &rest args
                        &key shelf tags title author &allow-other-keys
                        &aux (tags (alexandria:ensure-list tags)))
  (declare (ignore title author readedp)
           (type (shelf string)))
  (multiple-value-bind (book found?) (gethash designator *all-books*)
    (declare (ignore book))
    (if found?
        (error "Book with designator ~s already present." designator)
        (setf (gethash designator *all-books*)
              (make-instance 'book
                             :des designator
                             :shl shelf
                             :tgs (coerce tags 'list)
                             :meta args)))))

;;; Trivial
(defmethod delete-book ((designator symbol))
  (remhash designator *all-books*))

;;; We use `while-collecting' macro (`collect' equivalent from
;;; cmu-utils) to simplify the code.
(defmethod find-books (&rest args
                       &key
                         (book-designator nil designator-supplied-p)
                         (shelf nil shelf-supplied-p)
                         (tags nil tags-supplied-p)
                       &allow-other-keys
                       &aux (tags (alexandria:ensure-list tags)))
  (declare (ignore args))
  (uiop:while-collecting (match)
    (labels ((match-book (book)
               (and (or (null shelf-supplied-p)
                        (equalp shelf (book-shelf book)))
                    (or (null tags-supplied-p)
                        (subsetp tags (book-tags book) :test #'equalp))
                    (match book))))
      (if designator-supplied-p
          (alexandria:when-let ((book (gethash book-designator *all-books*)))
            (match-book book))
          (alexandria:maphash-values (lambda (val)
                                       (match-book val))
                                     *all-books*)))))

Our prototype support only shelf and tags filters and allows searching with a designator. Note that book-designator function is implemented in our class definition as a reader, so we don't have to define the method manually. We add uiop to dependencies for the while-collecting macro (descendant of a collect macro in cmu-utils).

We may check if our bare (without user interface) implementation works:

(ql:quickload :clamber)
;; -> (:CLAMBER)

(clamber:insert-book
 'captive-mind
 :shelf "nonfiction"
 :tags '("nonfiction" "politics" "psychology")
 :title "The Captive Mind"
 :author "Czesław Miłosz")
;; -> #<CLAMBER::BOOK {100469CB73}>

(clamber:find-books :tags '("politics"))
;; -> (#<CLAMBER::BOOK {100469CB73}>)

Unit tests

Now we will add some basic unit tests. For that we will use fiveam testing framework. For seamless integration with ASDF and to not include the tests in clamber itself we will define it as a separate system and point to it with the :in-order-to clause:

(asdf:defsystem #:clamber
  :description "Book collection managament."
  :author "Daniel Kochmański <daniel@turtleware.eu>"
  :license "Public Domain"
  :depends-on (#:alexandria #:uiop)
  :serial t
  :components ((:file "clamber"))
  :in-order-to ((asdf:test-op
                 (asdf:test-op #:clamber/tests))))

(asdf:defsystem #:clamber/tests
  :depends-on (#:clamber #:fiveam)
  :components ((:file "tests"))
  :perform (asdf:test-op (o s)
             (uiop:symbol-call :clamber/tests :run-tests)))

tests.lisp file is in the repository with clamber. To run the tests issue in the REPL:

(asdf:test-system 'clamber/tests)

Prototype data persistence

To make our prototype complete we need to store our database. We will use for it a directory returned by uiop:xdg-data-home. To serialize a hash-table cl-store will be used.

(defparameter *database-file* (uiop:xdg-data-home "clamber" "books.db"))

(defun restore-db ()
  "Restore a database from the file."
  (when (probe-file *database-file*)
    (setf *all-books* (cl-store:restore *database-file*))))

(defun store-db ()
  "Store a database in the file."
  (ensure-directories-exist *database-file*)
  (cl-store:store *all-books* *database-file*))

(defmethod insert-book :around ((designator symbol)
                                &rest args &key &allow-other-keys)
  (declare (ignore designator args))
  (prog2 (restore-db)
      (call-next-method)
    (store-db)))

(defmethod delete-book :around ((designator symbol))
  (declare (ignore designator))
  (prog2 (restore-db)
      (call-next-method)
    (store-db)))

(defmethod find-books :around (&rest args &key &allow-other-keys)
  (declare (ignore args))
  (restore-db)
  (call-next-method))

We read and write database during each operation (not very efficient, but it is just a prototype). find-books doesn't need to store the database, because it doesn't modify it.

Since our database isn't only in-memory object anymore, some additional changes to tests seem appropriate. We don't want to modify user's database:

(defparameter *test-db-file*
  (uiop:xdg-data-home "clamber" "test-books.db"))

(defun run-tests ()
  (let ((clamber::*database-file* *test-db-file*))
    (5am:run! 'clamber)))

Right now we have a "working" prototype, what we need is the user interface.

Creating standalone executable

There are various solutions which enable creation of standalone binaries. The most appealing to me is Clon: the Command-Line Options Nuker, which has a very complete documentation (end-user manual, user manual and reference manual) , well thought API and works on a wide range of implementations. Additionally, it is easy to use and covers various corner-cases in a very elegant manner.

Our initial CLI (Command Line Interface) will be quite modest:

% clamber --help
% clamber add-book foo \
  --tags a,b,c \
  --shelf "Favourites" \
  --meta author "Bar" title "Quux"  
% clamber del-book bah
% clamber list-books
% clamber list-books --help
% clamber list-books --shelf=bah --tags=drama,psycho
% clamber show-book bah

Basic CLI interface

To make our interface happen we have to define application synopsis. clon provides defsynopsis macro for that purpose:

(defsynopsis (:postfix "cmd [OPTIONS]")
  (text :contents
        "Available commands: add-book, del-book, list-books, show-book.
Each command has it's own `--help' option.")
  (flag :short-name "h" :long-name "help"
        :description "Print this help and exit.")
  (flag :short-name "g" :long-name "gui"
        :description "Use graphical user interface."))

These are all top-level flags handling main options (help and graphical mode switch). As we can see it is declarative, allowing short and long option names. Except flag other possible option types are possible (user may even add his own kind of option).

clon allows having multiple command line option processing contexts, what simplifies our task – we can provide different synopsis for each command with its own help. First though we will define a skeleton of our main function:

(defun main ()
  "Entry point for our standalone application."
  ;; create default context
  (make-context)
  (cond
    ;; if user asks for help or invokes application without parameters
    ;; print help and quit
    ((or (getopt :short-name "h")
         (not (cmdline-p)))
     (help)
     (exit))
    ;; running in graphical mode doesn't require processing any
    ;; further options
    ((getopt :short-name "g")
     (print "Running in graphical mode!")
     (exit)))

  (alexandria:switch ((first (remainder)) :test 'equalp)
    ("add-book"   (print "add-book called!"))
    ("del-book"   (print "del-book called!"))
    ("list-books" (print "list-books called!"))
    ("show-book"  (print "show-book called!")))

  (exit))

(defun dump-clamber (&optional (path "clamber"))
  (dump path main))

In our main we look for the top-level options first. After that we verify which command is called. For now our action is just a stub which prints the command name. We will expand it in the next step. Function dump-clamber is provided to simplify executable creation. To dump the executable it is enough to use this snippet:

sbcl --eval '(ql:quickload :clamber)' --eval '(clamber/cli:dump-clamber "clamber")'
./clamber --help

img

Implementing commands

Each command has to have its own synopsis. Books have unique identifiers (designators) – we force this option to be a symbol. All applications parameters following the options are treated as metadata. add-book has the following synopsis:

(defparameter +add-book-synopsis+
  (defsynopsis (:make-default nil :postfix "cmd [OPTIONS] [META]")
    (text :contents "Add a new book to the database.")
    (flag :short-name "h" :long-name "help"
          :description "Print this help and exit.")
    (lispobj :short-name "d" :long-name "ident"
             :description "Book designator (unique)."
             :typespec 'symbol)
    (stropt :short-name "s" :long-name "shelf"
            :description "Book shelf.")
    ;; comma-separated (no spaces)
    (stropt :short-name "t" :long-name "tags"
            :description "Book tags."))
  "The synopsis for the add-book command.")

We don't want duplicated options, so we filter them out in the add-book-main function, which is called in main instead of printing the message. Command entry point is implemented as follows:

(defun add-book-main (cmdline)
  "Entry point for `add-book' command."
  (make-context :cmdline cmdline
                :synopsis +add-book-synopsis+)
  (when (or (getopt :short-name "h")
            (not (cmdline-p)))
    (help)
    (exit))

  (let ((ident (getopt :short-name "d"))
        (shelf (getopt :short-name "s"))
        (tags  (getopt :short-name "t")))

    (when (or (getopt :short-name "d")
              (getopt :short-name "s")
              (getopt :short-name "t"))
      (print "add-book: Junk on the command-line.")
      (exit 1))
    (clamber:insert-book ident
                         :shelf shelf
                         :tags (split-sequence #\, tags)
                         :meta (remainder))))

To make book listing more readable we define print-object method for books in clamber.lisp. Moreover, we tune find-books method to not rely on the fact whenever argument was supplied or not, but rather on its value (NIL vs. non-NIL).

(defmethod print-object ((object book) stream)
  (if (not *print-escape*)
      (format stream "~10s [~10s] ~s -- ~s"
              (book-designator object)
              (book-shelf object)
              (book-tags object)
              (book-information object))
      (call-next-method)))

list-books command is very similar, but instead of calling insert-book it prints all books found with clamber:find-books called with provided arguments. Also we don't print help if called without any options.

(defparameter +list-books-synopsis+
  (defsynopsis (:make-default nil :postfix "[META]")
    (text :contents "List books in the database.")
    (flag :short-name "h" :long-name "help"
          :description "Print this help and exit.")
    (lispobj :short-name "d" :long-name "ident"
             :description "Book designator (unique)."
             :typespec 'symbol)
    (stropt :short-name "s" :long-name "shelf"
            :description "Book shelf.")
    ;; comma-separated (no spaces)
    (stropt :short-name "t" :long-name "tags"
            :description "Book tags."))
  "The synopsis for the list-books command.")

(defun list-books-main (cmdline)
  "Entry point for `list-books' command."
  (make-context :cmdline cmdline
                :synopsis +list-books-synopsis+)
  (when (getopt :short-name "h")
    (help)
    (exit))

  (let ((ident (getopt :short-name "d"))
        (shelf (getopt :short-name "s"))
        (tags  (getopt :short-name "t")))

    (when (or (getopt :short-name "d")
              (getopt :short-name "s")
              (getopt :short-name "t"))
      (print "add-book: Junk on the command-line.")
      (exit 1))

    (map () (lambda (book)
              (format t "~a~%" book))
         (clamber:find-books :book-designator ident
                             :shelf shelf
                             :tags tags))))

Last command we are going to implement is the simplest one – del-book:

(defparameter +del-book-synopsis+
  (defsynopsis (:make-default nil)
    (text :contents "Delete a book in the database.")
    (flag :short-name "h" :long-name "help"
          :description "Print this help and exit.")
    (lispobj :short-name "d" :long-name "ident"
             :description "Book designator (unique)."
             :typespec 'symbol))
  "The synopsis for the del-book command.")

(defun delete-book-main (cmdline)
  "Entry point for `list-books' command."
  (make-context :cmdline cmdline
                :synopsis +del-book-synopsis+)
  (when (or (getopt :short-name "h")
            (not (cmdline-p)))
    (help)
    (exit))

  (clamber:delete-book (getopt :short-name "d")))

Of course this CLI prototype needs to be improved. For instance, it doesn't handle any errors – for if we try to add a book with already existing designator. Moreover, for testing purposes it would be nice to be able to provide database file top-level argument for testing purposes.

Footnotes:

1
Since we won't use any unique `ECL` features we suggest using `SBCL` here (it is faster and better supported by 3rd-party libraries). Using `ECL` shouldn't introduce any problems though.