Rework ABCL-BUILD as a contrib

ABCL-BUILD now more usefully installs necessary build machinery.

Refactor the ABCL-BUILD Lisp based build to use use Ant, which allows
us to canonicalize the build instructions in a much more configurable
manner, restoring the cross-implementation build of ABCL for CL users
who really do want to know about Java tooling.

Since we want to use ASDF/UIOP anyways, this code is best expressed as
an abcl contrib where such behavior can profitably take place.

TODO: does not pass all tests

TODO: API compatibility so that build-from-lisp.bash works

TODO: If an invocable Ant cannot be found, then offer to install one via restart

TODO: work backwards to the scenario that we are booting in pure
interepreted mode without access to compilers.
parent 42989ed4
...@@ -130,7 +130,7 @@ preferred (and most tested way) is to being to use the Ant build tool: ...@@ -130,7 +130,7 @@ preferred (and most tested way) is to being to use the Ant build tool:
implementations for this process: SBCL, CMUCL, OpenMCL, Allegro implementations for this process: SBCL, CMUCL, OpenMCL, Allegro
CL, LispWorks or CLISP. CL, LispWorks or CLISP.
In all cases you need a Java 5 or later JDK (JDK 1.[5678] have been In all cases you will need a Java 6 or later JDK (JDK 1.[678] have been
tested). Just the JRE isn't enough, as you need the Java compiler tested). Just the JRE isn't enough, as you need the Java compiler
('javac') to compile the Java source of the ABCL implementation. ('javac') to compile the Java source of the ABCL implementation.
...@@ -178,21 +178,9 @@ debug process. ...@@ -178,21 +178,9 @@ debug process.
Building from Lisp Building from Lisp
------------------ ------------------
Building from another Common Lisp implementation is the most venerable You need to have a Java compiler installed locally. If the Ant build
and least tested way of building ABCL. It produces a "non-standard" tool does not exist, it will be downloaded and used to invoke the Ant
version of the distribution that doesn't share build instructions with build.
the previous two methods, but it still may be of interest to those who
absolutely don't want to know anything about Java. The other Common
Lisp implementation does not actually perform any compliation, but is
merely used to orchestrate the invocation of tools in the JDK.
First, copy the file 'customizations.lisp.in' to 'customization.lisp',
in the directory containing this README file, editing to suit your
situation, paying attention to the comments in the file. The critical
step is to have Lisp special variable '*JDK*' point to the root of the
Java Development Kit. Underneath the directory referenced by the
value of '*JDK*' there should be an executable Java compiler in
'bin/javac' ('bin/javac.exe' under Windows).
Then, one may either use the 'build-from-lisp.bash' shell script or load Then, one may either use the 'build-from-lisp.bash' shell script or load
the necessary files into your Lisp image by hand. the necessary files into your Lisp image by hand.
...@@ -229,7 +217,7 @@ into limitations on command line length (but is a lot slower). ...@@ -229,7 +217,7 @@ into limitations on command line length (but is a lot slower).
### Building from another Lisp manually ### Building from another Lisp manually
There is also an ASDF definition in 'abcl.asd' for the BUILD-ABCL There is also an ASDF definition in 'abcl.asd' for ABCL/BUILD
which can be used to load the necessary Lisp definitions, after which which can be used to load the necessary Lisp definitions, after which
CL-USER> (build-abcl:build-abcl :clean t :full t) CL-USER> (build-abcl:build-abcl :clean t :full t)
......
...@@ -138,42 +138,6 @@ be in a directory named '../ansi-test/'." ...@@ -138,42 +138,6 @@ be in a directory named '../ansi-test/'."
(:module wrapper :pathname "test/lisp/cl-bench/" (:module wrapper :pathname "test/lisp/cl-bench/"
:depends-on (package) :components :depends-on (package) :components
((:file "wrapper"))))) ((:file "wrapper")))))
;;; aka the "Lisp-hosted build system" which doesn't share build
;;; instructions with the canonical build system in <file:build.xml>
;;; Works for: abcl, sbcl, clisp, cmu, lispworks, allegro, openmcl
(defsystem abcl/build
:version "2.0.0"
:description "Build ABCL from a Lisp. Downloads necessary build-time tools to local cache."
:in-order-to ((test-op (test-op abcl/build/t)))
:components ((:module package
:pathname "src/org/abcl/lisp/build/"
:components ((:file "package")))
(:module build
:pathname "src/org/abcl/lisp/build/"
:depends-on (package)
:serial t
:components (;;; TODO optionally parse a local configuration for customization
(:file "platform")
(:file "customizations-default")
(:file "install")
(:file "ant")
(:file "build-abcl")))))
(defsystem abcl/build/t
:description "Test ABCL build system."
:defsystem-depends-on (prove-asdf)
:depends-on (abcl/build
prove)
:perform (asdf:test-op (op c)
(uiop:symbol-call :prove-asdf 'run-test-system c))
:components ((:module package
:pathname "t/"
:components ((:file "package")))
(:module build
:depends-on (package)
:pathname "t/"
:components ((:test-file "abcl-build")))))
(defsystem abcl/documentation (defsystem abcl/documentation
:description "Tools to generate LaTeX source from docstrings." :description "Tools to generate LaTeX source from docstrings."
:depends-on (swank) :depends-on (swank)
......
...@@ -5,11 +5,10 @@ ...@@ -5,11 +5,10 @@
usage() usage()
{ {
echo "WARNING: This build method can produce slightly different results than"
echo " the canonical Ant build method."
echo echo
echo "USAGE:" echo "USAGE:"
echo "$0 <implementation> [[ --clean=T | --full=T | --batch=NIL ]]" echo "$0 <implementation>"
# echo "$0 <implementation> [[ --clean=T | --full=T | --batch=NIL ]]"
} }
if [ -z "$1" ]; then if [ -z "$1" ]; then
...@@ -39,10 +38,12 @@ if [ $? != 0 ] ; then ...@@ -39,10 +38,12 @@ if [ $? != 0 ] ; then
fi fi
eval set -- "$TEMP" eval set -- "$TEMP"
# TODO all ignored
CLEAN="t" CLEAN="t"
FULL="t" FULL="t"
BATCH="t" BATCH="t"
# TODO all ignored
while true ; do while true ; do
case "$1" in case "$1" in
--clean) --clean)
...@@ -65,21 +66,8 @@ while true ; do ...@@ -65,21 +66,8 @@ while true ; do
esac esac
done done
FORM="(build-abcl:build-abcl :clean $CLEAN :full $FULL :batch $BATCH)" FORM="(abcl/build:abcl/build)"
FILE="build-abcl.lisp" FILE="src/org/abcl/lisp/build/build-abcl.lisp"
if [[ ! -r customizations.lisp ]]; then
cat <<-EOF
Failed to find configuration in 'customizations.lisp'. Please copy
'customizations.lisp.in' to 'customizations.lisp', editing it to
reflect local configuration.
EOF
exit 1
fi
abcl() abcl()
{ {
......
...@@ -79,10 +79,6 @@ For help on the automatic tests available, use the Ant target 'help.test'. ...@@ -79,10 +79,6 @@ For help on the automatic tests available, use the Ant target 'help.test'.
<exclude name="org/armedbear/lisp/j.lisp"/> <exclude name="org/armedbear/lisp/j.lisp"/>
<include if="abcl.jsr-223.p" name="org/armedbear/lisp/scripting/lisp/*.lisp"/> <include if="abcl.jsr-223.p" name="org/armedbear/lisp/scripting/lisp/*.lisp"/>
</patternset> </patternset>
<patternset id="abcl.source.lisp.build">
<include name="org/abcl/lisp/build/*.lisp"/>
</patternset>
<!-- Lisp files required at runtime --> <!-- Lisp files required at runtime -->
<patternset id="abcl.source.lisp.dist"> <patternset id="abcl.source.lisp.dist">
...@@ -210,8 +206,6 @@ For help on the automatic tests available, use the Ant target 'help.test'. ...@@ -210,8 +206,6 @@ For help on the automatic tests available, use the Ant target 'help.test'.
<src path="${src.dir}"/> <src path="${src.dir}"/>
<patternset refid="abcl.source.java"/> <patternset refid="abcl.source.java"/>
</javac> </javac>
<echo message="${build}"
file="${build.classes.dir}/org/armedbear/lisp/build"/>
</target> </target>
<!-- Additional artifacts to stage relative to Ant ${basedir} --> <!-- Additional artifacts to stage relative to Ant ${basedir} -->
...@@ -233,9 +227,6 @@ For help on the automatic tests available, use the Ant target 'help.test'. ...@@ -233,9 +227,6 @@ For help on the automatic tests available, use the Ant target 'help.test'.
<fileset dir="${basedir}/"> <fileset dir="${basedir}/">
<patternset refid="abcl.stage"/> <patternset refid="abcl.stage"/>
</fileset> </fileset>
<fileset dir="${src.dir}">
<patternset refid="abcl.source.lisp.build"/>
</fileset>
</copy> </copy>
</target> </target>
......
...@@ -65,11 +65,16 @@ abcl-introspect ...@@ -65,11 +65,16 @@ abcl-introspect
Advanced introspection of Java and Lisp runtime classes Advanced introspection of Java and Lisp runtime classes
representation. representation.
abcl-build
The ABCL build system plus associated utilities for manipulating
path executables
# Colophon # Colophon
Mark <evenson.not.org@gmail.com> Mark <evenson.not.org@gmail.com>
Created: 2011-09-11 Created: 2011-09-11
Revised: 2016-12-28 Revised: 2017-05-22
<> abcl:documents <abcl.org/release/1.5.0/#abcl-contrib.jar> . <> abcl:documents <abcl.org/release/1.5.0/#abcl-contrib.jar> .
......
ABCL-BUILD
==========
Installing the necessary toolchain to build ABCL.
Plus some utility functions to download needed artifacts from the
network.
;;; aka the "Lisp-hosted build system" which doesn't share build
;;; instructions with the canonical build system in <file:build.xml>
;;; Works for: abcl, sbcl, clisp, cmu, lispworks, allegro, openmcl
(defsystem abcl-build
:version "2.0.0"
:description "Build ABCL from a Lisp. Downloads necessary build-time tools to local cache."
:in-order-to ((test-op (test-op abcl-build/t)))
:components ((:module package
:pathname "build/"
:components ((:file "package")))
(:module util
:pathname "build/"
:depends-on (package)
:components ((:file "util")))
(:module build
:pathname "build/"
:depends-on (util)
:serial t
:components (;;; TODO optionally parse a local configuration for customization
(:file "customizations-default")
(:file "install")
(:file "maven")
(:file "ant")
(:file "abcl-build") ;; TODO: support API
(:file "deprecated")))))
(defsystem abcl-build/t
:description "Test ABCL build system."
:defsystem-depends-on (prove-asdf)
:depends-on (abcl-build
prove)
:perform (asdf:test-op (op c)
(uiop:symbol-call :prove-asdf 'run-test-system c))
:components ((:module package
:pathname "build/t/"
:components ((:file "package")))
(:module build
:depends-on (package)
:pathname "build/t/"
:components ((:test-file "util")
(:test-file "install")
(:test-file "ant")
(:test-file "maven")
(:test-file "abcl-build")))))
(in-package :abcl/build)
(defun make-dist (version-string)
(warn "Unimplemented"))
(defun build-abcl (&key force
(batch t)
compile-system
jar
clean
full)
(warn "Unimplemented"))
(in-package :abcl/build)
;; TODO function to deal with looking up a locally preferred mirrors
(defun ant-zip-uri ()
#p"http://archive.apache.org/dist/ant/binaries/apache-ant-1.9.4-bin.zip"
#+nil ;; https on OPEN fails; probably attempting to upgrade
#p"https://archive.apache.org/dist/ant/binaries/apache-ant-1.9.4-bin.zip"
#+nil ;; need apache-ant-1.9 for JVM version 49.0
#p"http://www-eu.apache.org/dist/ant/binaries/apache-ant-1.10.1-bin.zip")
(defun xdg/ant-executable ()
(probe-file
(localize-executable-name
(merge-pathnames #p"bin/ant"
(xdg/abcl-install-root (ant-zip-uri))))))
(defun ant/install ()
(unless (xdg/ant-executable)
(xdg/install (ant-zip-uri) :type :unzip))
(values
(xdg/ant-executable)
(directory (merge-pathnames "**/*"
(xdg/abcl-install-root (ant-zip-uri))))))
(defparameter *ant-home* nil)
(define-condition no-installed-ant (error)
((searched))
(:report (lambda (condition stream)
(format stream "Unable to introspect Apache Ant installation."))))
;; TODO after this routines executes *ANT-EXECUTABLE-DIRECTORY* and XDG/ANT-EXECUTABLE will work
(defun ensure-ant (&key (ant-home nil ant-home-p))
(cond
((and (null ant-home) ant-home-p)
(warn "Unimplemented explicit auto-configuration run."))
((and ant-home ant-home-p)
(warn "Unimplemented explicit configuration with specified directory directory."))
(t
(if *ant-home*
*ant-home*
(restart-case
(let ((ant-home (some-directory-containing "ant")))
(unless ant-home
(signal 'no-installed-ant))
(setf *ant-home ant-home))
(install-ant ()
(ant/install)))))))
(defmacro with-ensured-ant ((ant) &body body)
`(progn
(unless ,ant
(setf ,ant (ensure-ant)))
,@body))
(defun ant/call (ant-file target-or-targets)
(let (ant)
(with-ensured-ant (ant)
(uiop:run-program
`(,ant "-buildfile"
,(stringify ant-file)
,@(listify target-or-targets))
:output :string))))
(in-package :abcl/build)
(defun abcl/build ()
(abcl-build:ant/call (asdf:system-relative-pathname :abcl "build.xml")
"abcl"))
(defun abcl/dist ()
(abcl-build:ant/call (asdf:system-relative-pathname :abcl "build.xml")
"abcl.release"))
(defun abcl/test ()
(abcl-build:ant/call (asdf:system-relative-pathname :abcl "build.xml")
"abcl.test"))
...@@ -17,10 +17,22 @@ ...@@ -17,10 +17,22 @@
;;; Directories should be specified with a trailing slash (or, on Windows, a ;;; Directories should be specified with a trailing slash (or, on Windows, a
;;; trailing backslash). ;;; trailing backslash).
(in-package "BUILD-ABCL") (in-package :abcl/build)
;; Standard compiler options. ;; Standard compiler options.
(setq *javac-options* "-g") (defparameter *javac-options*
(setq *jikes-options* "+D -g") "-g")
(defparameter *jikes-options*
"+D -g")
(defparameter *jdk*
(cond
((uiop:os-macosx-p)
"/usr/")
(t
(introspect-path-for "javac"))))
#-abcl (error "Sorry, but this only currently works with the Bear.")
(in-package :abcl/build)
(defun xdg/abcl-install-root (uri)
"Return the private xdg rooted installation location for URI."
(merge-pathnames
(make-pathname :directory `(:relative "abcl" "install" ,(pathname-name uri)))
(uiop/configuration:xdg-data-home)))
(defun xdg/abcl-download-root (&key (for-uri nil for-uri-p))
(declare (ignore for-uri-p))
(let ((root (merge-pathnames
(make-pathname :directory '(:relative "abcl" "dist"))
(uiop/configuration:xdg-data-home)))) ;; TODO move to proper XDG cache hierarchy
(unless for-uri
(return-from xdg/abcl-download-root root))
(let* ((uri (if (pathnamep for-uri)
for-uri
(pathname for-uri)))
(name (pathname-name uri)))
(merge-pathnames
(make-pathname :directory `(:relative ,name))
root))))
(defgeneric xdg/install ((uri pathname) &key type)
(:method ((uri pathname) &key (type :unzip))
(declare (ignore type))
(download-and-unzip uri)))
(defun download-and-unzip (uri)
(let ((archive
(download uri))
(root
(xdg/abcl-download-root :for-uri uri)))
(ensure-directories-exist root)
(sys:unzip archive root)
(values
root
(directory (merge-pathnames "**/*" root)))))
(defun download (uri &key (destination
(merge-pathnames
(make-pathname :defaults uri :host nil :device nil :directory nil)
(xdg/abcl-download-root))))
"Download the contents of URI to DESTINATION.
Returns the local pathname of the download artifact."
(ensure-directories-exist destination)
(uiop:copy-file
(open uri :direction :input)
destination)
destination)
(in-package :abcl/build)
(defun maven-zip-uri ()
#p"http://www-eu.apache.org/dist/maven/maven-3/3.5.0/binaries/apache-maven-3.5.0-bin.zip")
(defun xdg/mvn-executable ()
(probe-file
(localize-executable-name
(merge-pathnames "bin/mvn"
(xdg/abcl-install-root (maven-zip-uri))))))
(defparameter *maven-install-root* nil)
(defun mvn/install ()
(unless (xdg/mvn-executable)
(xdg/install (maven-zip-uri) :type :unzip))
(values
(xdg/mvn-executable)
(directory (merge-pathnames
"**/*" (xdg/abcl-install-root (maven-zip-uri))))))
(defparameter *mvn-home* nil)
(define-condition no-installed-maven (error)
((searched :initarg :searched))
(:report (lambda (condition stream)
(declare (ignore condition))
(format stream "Unable to introspect local Apache Maven installation."))))
(defun ensure-maven (&key (mvn-home *mvn-home* mvn-home-p))
(cond
((and (null mvn-home) mvn-home-p)
(warn "Unimplemented explicit auto-configuration run."))
((and mvn-home mvn-home-p)
(warn "Unimplemented explicit configuration with specified directory directory."))
(t
(if *mvn-home*
*mvn-home*
(restart-case
(let ((mvn-home (some-directory-containing "mvn")))
(unless mvn-home
(signal 'no-installed-maven))
(setf *mvn-home* mvn-home))
(install-maven ()
(mvn/install)))))))
(defmacro with-ensured-mvn ((maven) &body body)
`(progn
(unless ,maven
(setf ,maven (ensure-maven))
,@body)))
(defun mvn/call (pom-file target-or-targets)
(let (mvn)
(with-ensured-mvn (mvn)
(uiop:run-program
`(,mvn "--file" ,(stringify pom-file)
,@(listify target-or-targets))
:output :string))))
(in-package :cl-user)
(defpackage build-abcl
(:use :cl)
(:nicknames :build-abcl :abcl-build :abcl/build)
(:export
#:abcl/build
#:abcl/dist
#:abcl/test
;; deprecated TODO: hook into new interfaces
#:build-abcl
#:make-dist
;; utility functions that should be moved into utility package
#:introspect-path-for
#:split-string
#:localize-executable-name
#:stringify
#:listify
#:some-directory
;;; lower-level
#:xdg/abcl-install-root
#:xdg/abcl-download-root
#:xdg/install
#:locally-install-and-unzip
#:download-and-unzip
#:download
#:with-ensured-ant
#:ant/install
#:ant/call
#:with-ensured-maven
#:mvn/install
#:mvn/call
#:install-zip
#:download-artifact)
;;; TODO: use UIOP, currently only used for deprecated, old build system
#+abcl
(:import-from #:extensions #:run-shell-command #:probe-directory)
#+allegro
(:import-from #:excl #:probe-directory)
#+clisp
(:import-from #:ext #:probe-directory))
(in-package :abcl/build/t)
(prove:plan 2)
(prove:diag "Testing BUILD-ABCL…")
(prove:ok
(abcl-build:build-abcl))
(prove:diag "Testing MAKE-DIST…")
(prove:ok
(abcl-build:make-dist (format nil "test-" (random (expt 2 32)))))
#+abcl-build-test-more
(progn
(prove:plan 1)
(prove:ok (abcl-build:build-abcl :clean t)))
#+abcl-build-test-more
(progn
(prove:plan 1)
(abcl-build:build-abcl :force t))
(prove:finalize)
(in-package :abcl/build/t)
(prove:plan 1)
(prove:diag "Testing private installation of Ant…")
(prove:ok (build-abcl:ant/install))
(prove:plan 1)
(let ((antfile (asdf:system-relative-pathname :abcl "build.xml")))
(prove:diag (format nil "Testing invocation of private Ant on main ABCL build artifact at ~&~2,t~a…"
antfile))
(prove:ok (abcl-build:ant/call antfile "abcl.diagnostic")))
(prove:finalize)
(in-package :cl-user)
(prove:plan 1)
(prove:ok
(abcl/build:xdg/abcl-install-root #P"http://archive.apache.org/dist/ant/binaries/apache-ant-1.9.4-bin.zip"))
(prove:plan 2)
(let ((path (ext:make-temp-directory))
(uri #p"http://www-eu.apache.org/dist/ant/binaries/apache-ant-1.10.1-bin.zip"))
(prove:diag
(format nil "Testing binary unzip installation of~%~,2t<~a>~%to~%~,2t '~a'." uri path))
(multiple-value-bind (root contents)
(abcl/build:xdg/install uri)
(prove:ok (and root
(probe-file root)))
(prove:ok (and
(consp contents)
(> (length contents) 0)))))
(prove:finalize)
(in-package :abcl/build/t)
(prove:plan 1)
(prove:diag "Testing private installation of Maven…")
(prove:ok
(abcl/build:mvn/install))
(prove:plan 1)
(let ((pom (asdf:system-relative-pathname :abcl "pom.xml")))
(prove:diag (format nil "Testing invocation of private Maven on root ABCL POM at~&~2,t~a…"
pom))
(prove:ok (abcl/build:mvn/call pom "install")))
(prove:finalize)
(defpackage :abcl/build/t
(:use :cl))
(in-package :abcl/build/t)
(let ((executable
(abcl/build:localize-executable-name "java"))) ;; RUNTIME difference
(prove:plan 1)
(prove:ok executable)
(if (uiop:os-windows-p)
(prove:plan 1)
(prove:is (pathname-type executable) "exe")))
(prove:plan 1)
(prove:is (length (abcl/build:split-string "one.two.three." #\.))
3)
(prove:finalize)
;;;; TODO: move to a utility package
(in-package :abcl/build)
(defun localize-executable-name (name)
(let* ((p (if (pathnamep name)
name
(pathname name)))
(type (pathname-type p)))
(make-pathname :defaults p
:type
(if (uiop:os-windows-p)
(when (null type)
"exe")
type))))
(defun introspect-path-for (executable)
(let ((which-command (if (uiop:os-windows-p)
"where"
"which")))
(when (ignore-errors
(uiop:run-program (list which-command which-command) :output :string))
(let ((raw-result
(uiop:run-program
(list which-command
(namestring (localize-executable-name executable)))
:output :string)))
(let ((result
(string-trim
'(#\Space #\Newline #\Backspace #\Tab
#\Linefeed #\Page #\Return #\Rubout)
raw-result)))