From 61208c78bc75905bb29a7555938139cd17f0896c Mon Sep 17 00:00:00 2001 From: toy <toy> Date: Fri, 24 Oct 2003 16:55:36 +0000 Subject: [PATCH] Initial version. --- BUILDING | 501 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 501 insertions(+) create mode 100644 BUILDING diff --git a/BUILDING b/BUILDING new file mode 100644 index 000000000..92cc70993 --- /dev/null +++ b/BUILDING @@ -0,0 +1,501 @@ +-*- Mode: text -*- +Building CMU CL +=============== + +This document is intended to give you a general overview of the build +process (i.e. what needs to be done, in what order, and what is it +generally called). It will also tell you how to set up a suitable +build environment, how the individual scripts fit into the general +scheme of things, and give you a couple of examples. + +General Requirements +-------------------- + +In order to build CMU CL, you will need: + +a) A working CMU CL binary. There is no way around this requirement! + + This binary can either be for the platform you want to target, in + that case you can either recompile or cross-compile, or for another + supported platform, in that case you must cross-compile, obviously. + +b) A supported C compiler for the C runtime code. + + Most of the time, this means GNU gcc, though for some ports it + means the vendor-supplied C compiler. The compiler must be + available under the name specified by your ports Config file. + +c) GNU make + + This has to be available either as gmake or make in your PATH, or + the MAKE environment variable has to be set to point to the correct + binary. + +d) The CMU CL source code + + Here you can either use one of the release source tarballs, or + check out the source code directly from the public CMUCL CVS + repository. + +If you want to build CMU CL's Motif interface/toolkit, you'll need a +working version of the Motif libraries, either true-blue OSF/Motif, or +OpenMotif, or Lesstif. The code was developed against 1.2 Motif, +though recompilation against 2.x Motif probably works as well. + +Setting up a build environment +------------------------------ + +1.) Create a base directory and change to it + + mkdir cmucl ; cd cmucl + +2.) Fetch the sources and put them into the base directory + + tar xzf /tmp/cmucl-18d.source.tar.gz + + or, if you want to use the CVS sources directly: + + export CVSROOT=:pserver:anonymous@cvs2.cons.org:/home/anoncvs/CVS-cmucl + cvs login (password is `anonymous') + cvs co src + + Whatever you do, the sources must be in a directory named src + inside the base directory. Since the build tools keep all + generated files in separate target directories, the src directory + can be read-only (e.g. mounted read-only via NFS, etc.) + + The build tools are all in the src/tools directory. + +That's it, you are now ready to build CMU CL. + + +A general outline of the build process +-------------------------------------- + +Building CMU CL can happen in one of two ways: Normal recompilation, +and cross-compilation. We'll first look at normal recompilation: + +The recompilation process basically consists of 4 phases/parts: + +a) Compiling the lisp files that make up the standard kernel. + + This happens in your current CMU CL process, using your current + CMU CL's normal file compiler. This phase currently consists of 3 + sub-phases, namely those controlled by src/tools/worldcom.lisp, + which compiles all the runtime files, src/tools/comcom.lisp, which + compiles the compiler (including your chosen backend), and finally + src/tools/pclcom.lisp, which compiles PCL, CMU CL's CLOS + implementation. The whole phase is often called "world-compile", + or "compiling up a world", based on the name of the first + sub-phase. + +b) Building a new kernel.core file out of the so created files + + This process, which is generally called genesis, and which is + controlled by src/tools/worldbuild.lisp, uses the newly compiled + files in order to build a new, basic core file, which is then used + by the last phase to create a fully functional normal core file. + It does this by "loading" the compiled files into an in-core + representation of a new core file, which is then dumped out to + disk, together with lots of fixups that need to happen once the new + core is started. + + As part of this process, it also creates the file internals.h, + which contains information about the general memory layout of the + new core and its basic types, their type tags, and the location of + several important constants and other variables, that are needed by + the C runtime code to work with the given core. + + So going through genesis is needed to create internals.h, which is + needed to compile the C runtime code (i.e. the "lisp" binary). + However there is a slight circularity here, since genesis needs as + one of its inputs the file target:lisp/lisp.nm, which contains the + (slightly pre-treated) output of running nm on the new lisp + binary. Genesis uses this information to fixup the addresses of C + runtime support functions for calls from Lisp code. + + However the circularity isn't complete, since genesis can work with + an empty/bogus lisp.nm file. While the kernel.core it then + produces is unusable, it will create a usable internals.h file, + which can be used to recompile the C runtime code, producing a + usable lisp.nm file, which in turn can be used to restart genesis, + producing a working kernel.core file. + + Genesis also checks whether the newly produced internals.h file + differs from a pre-existing internals.h file (this might be caused + by an empty internals.h file if you are rebuilding for the first + time, or by changes in the lisp sources that cause differences in + the memory layout of the kernel.core), and informs you of this, so + that you can recompile the C runtime code, and restart genesis. + + If it doesn't inform you of this, you can skip directly to the last + phase d). + +c) Recompiling the C runtime code, producing the "lisp" binary file + + This step is only needed if you haven't yet got a suitable lisp + binary, or if the internals.h file has changed during genesis (of + which genesis informs you), or when you made changes to the C + sources that you want to take effect. + + Recompiling the C runtime code is controlled by a GNU Makefile, and + your target's Config file. It depends on a correct internals.h + file as produced by genesis. + + Note that whenever you recompile the runtime code, for whatever + reason, you must redo phase b). Note that if you make changes to + the C sources and recompile because of this, you can do that before + Phase b), so that you don't have to perform that phase twice. + +d) Populating the kernel.core, and dumping a new lisp.core file. + + In this phase, which is controlled by src/tools/worldload.lisp, and + hence often called world-load, the kernel.core file is started up + using the (possibly new) lisp binary, the remaining files which + were compiled in phase a) are loaded into it, and a new lisp.core + file is dumped out. + +When cross-compiling, there is additional phase at the beginning, and +some of the phases happen with different hosts/platforms. The initial +phase is setting up and compiling the cross-compilation backend, using +your current compiler. The new backend is then loaded, and all +compilation in phase a) happens using this compiler backend. The +creation of the kernel.core file in phase b) happens as usual, while +phase c) of course happens on the target platform (if that differs +from the host platform), as does the final phase d). Another major +difference is that you can't compile PCL using the cross-compiler, so +one usually does a normal rebuild using the cross-compiled core on the +target platform to get a full CMU CL core. + +So, now you know all about CMU CL compilation, how does that map onto +the scripts included with this little text? + +Overview of the included build scripts +-------------------------------------- + +* create-target.sh target-directory [lisp-variant [motif-variant]] + +This script creates a new target directory, which is a shadow of the +source directory, that will contain all the files that are created by +the build process. Thus, each target's files are completely separate +from the src directory, which could, in fact, be read-only. Hence you +can simultaneously build CMUCL for different targets from the same +source directory. + +The first argument is the name of the target directory to create. The +remaining arguments are optional. If they are not given, the script +tries to determine the lisp variant and motif variant from the system +the script is running on. + +The lisp-variant (i.e. the suffix of the src/lisp/Config.* to use as +the target's Config file), and optionally the motif-variant (again the +suffix of the src/motif/server/Config.* file to use as the Config file +for the target's CMUCL/Motif server code). If the lisp-variant is +given but the motif-variant is not, the motif-variant is determined +from the lisp-variant. + +The script will generate the target directory tree, link the relevant +Config files, and generate place-holder files for various files, in +order to ensure proper operation of the other build-scripts. It also +creates a sample setenv.lisp file in the target directory, which is +used by the build and load processes to set up the correct list of +*features* for your target lisp core. + +IMPORTANT: You will normally NOT have to modify the sample setenv.lisp +file, if you are building from a binary that has the desired features. +In fact, the sample has all code commented out, If you want to add or +remove features, you need to include code that puts at least a minimal +set of features onto the list (use PUSHNEW and/or REMOVE). You can +use the current set of *features* of your lisp as a first guide. The +sample setenv.lisp includes a set of features that should work for the +intended configuration. Note also that some adding or removing some +features may require a cross-compile instead of a normal compile. + +* clean-target.sh [-l] target-directory [more dirs] + +Cleans the given target directory, so that all created files will be +removed. This is useful to force recompilation. If the -l flag is +given, then the C runtime is also removed, including all the lisp +executable, any lisp cores, all object files, lisp.nm, internals.h, +and the config file. + +* build-world.sh target-directory [build-binary] [build-flags...] + +Starts a complete world build for the given target, using the lisp +binary/core specified as a build host. The recompilation step will +only recompile changed files, or files for which the fasl files are +missing. It will also not recompile the C runtime code (the lisp +binary). If a (re)compilation of that code is needed, the genesis +step of the world build will inform you of that fact. In that case, +you'll have to use the rebuild-lisp.sh script, and then restart the +world build process with build-world.sh + +* rebuild-lisp.sh target-directory + +This script will force a complete recompilation of the C runtime code +of CMU CL (aka the lisp executable). Doing this will necessitate +building a new kernel.core file, using build-world.sh. + +* load-world.sh target-directory version + +This will finish the CMU CL rebuilding process, by loading the +remaining compiled files generated in the world build process into the +kernel.core file, that also resulted from that process, creating the +final lisp.core file. + +You have to pass the version string as a second argument. The dumped +core will anounce itself using that string. Please don't use a string +consisting of an official release name only, (e.g. "18d"), since those +are reserved for official release builds. Including the build-date in +ISO8601 format is often a good idea, e.g. "18d+ 2002-05-06" for a +binary that is based on sources current on the 6th May, 2002, which is +post the 18d release. + +* build-utils.sh target-directory + +This script will build auxiliary libraries packaged with CMU CL, +including CLX, CMUCL/Motif, the Motif debugger, inspector, and control +panel, and the Hemlock editor. It will use the lisp executable and +core of the given target. + +* make-dist.sh [-bg] [-G group] [-O owner] target-directory version arch os + +This script creates both main and extra distribution tarballs from the +given target directory, using the make-main-dist.sh and +make-extra-dist.sh scripts. You can select the compression method via +the -b (bzip2) and -g (gzip) flags. The default is gzip compression. +When making the distribution, you can also select the group and owner +of the files via the -G and -O options. + +* make-main-dist.sh target-directory version arch os + +This script creates a main distribution tarball (both in gzipped and +bzipped variants) from the given target directory. This will include +all the stuff that is normally included in official release tarballs +such as lisp.core and the PCL libraries, including Gray streams and +simple streams. + +This is intended to be run from make-dist.sh. + +* make-extra-dist.sh target-directory version arch os + +This script creates an extra distribution tarball (both in gzipped and +bzipped variants) from the given target directory. This will include +all the stuff that is normally included in official extra release +tarballs, i.e. the auxiliary libraries such as CLX, CLM, and Hemlock. + +This is intended to be run from make-dist.sh. + + +* cross-build-world.sh target-directory cross-directory cross-script + [build-binary] [build-flags...] + +This is a script that can be used instead of build-world.sh for +cross-compiling CMUCL. In addition to the arguments of build-world.sh +it takes two further required arguments: The name of a directory that +will contain the cross-compiler backend (the directory is created if +it doesn't exist, and must not be the same as the target-directory), +and the name of a Lisp cross-compilation script, which is responsible +for setting up, compiling, and loading the cross-compiler backend. +The latter argument is needed because each host/target combination of +platform's needs slightly different code to produce a working +cross-compiler. + +We include a number of working examples of cross-compiler scripts in +the cross-scripts directory. You'll have to edit the features section +of the given scripts, to specify the features that should be removed +from the current set of features in the host lisp, and those that +should be added, so that the backend features are correct for the +intended target. + +You can look at Eric Marsden's collection of build scripts for the +basis of more cross-compiler scripts. + +Step-by-Step Example of recompiling CMUCL for OpenBSD +----------------------------------------------------- + +Set up everything as described in the setup section above. Then +execute: + +# Create a new target directory structure/config for OpenBSD: +src/tools/create-target.sh openbsd OpenBSD_gencgc OpenBSD + +# edit openbsd/setenv.lisp to contain what we want: +cat <<EOF > openbsd/setenv.lisp +;;; Put code to massage *features* list here... + +(in-package :user) + +(pushnew :openbsd *features*) +(pushnew :bsd *features*) +(pushnew :i486 *features*) +(pushnew :mp *features*) +(pushnew :hash-new *features*) +(pushnew :random-mt19937 *features*) +(pushnew :conservative-float-type *features*) +(pushnew :gencgc *features*) + +;;; Version tags + +(pushnew :cmu18d *features*) +(pushnew :cmu18 *features*) +(setf *features* (remove :cmu17 *features*)) +(setf *features* (remove :cmu18c *features*)) +EOF + +# Recompile the lisp world, and dump a new kernel.core: +src/tools/build-world.sh openbsd lisp # Or whatever you need to invoke your + # current lisp binary+core + +# If build-world tells you (as it will the first time) that: +# "The C header file has changed. Be sure to re-compile the startup +# code." +# You 'll need to start rebuild-lisp.sh to do that, and then reinvoke +# build-world.sh: + +# Recompile lisp binary itself: +src/tools/rebuild-lisp.sh openbsd + +# Restart build-world.sh now: +src/tools/build-world.sh openbsd lisp + +# Now we populate the kernel.core with further compiled files, +# and dump the final lisp.core file: + +src/tools/load-world.sh openbsd "18d+ 2002-05-06" + +# The second argument above is the version number that the built +# core will announce. Please always put the build-date and some +# other information in there, to make it possible to differentiate +# those builds from official builds, which only contain the release. + +Now you should have a new lisp.core, which you can start with + +./openbsd/lisp/lisp -core ./openbsd/lisp/lisp.core -noinit -nositeinit + +Compiling sources that contain disruptive changes +------------------------------------------------- + +The above instructions should always work as-is for recompiling CMU CL +using matching binaries and source files. They also work quite often +when recompiling newer sources. However, every so often, some change +to the CMU CL sources necessitates some form of bootstrapping, so that +binaries built from earlier sources can compile the sources containing +that change. There are two forms of boostrapping that can be +required: + +a) Bootfiles + + The maintainers try to make bootfiles available, that allow going + from an old release to the next release. These are located in the + src/bootfiles/<old-release>/ directory of the CMU CL sources. + + I.e. if you have binaries that match release 18d, then you'll need + to use all the bootfiles in src/bootfiles/18d/ in order to go to + the next release (or current sources, if no release has been made + yet). If you already used some of the bootstrap files to compile + your current lisp, you obviously don't need to use those to get to + later versions. + + You can use the bootfiles by concatenating them into a file called + bootstrap.lisp in the target directory (i.e. target:bootstrap.lisp) + in the order they are numbered. Be sure to remove the bootstrap + file once it is no longer needed. + + +b) Cross-compiling + + Under some circumstances, bootstrap code will not be sufficient, + and a cross-compilation is needed. In that case you will have to + use cross-build-world.sh, instead of build-world.sh. Please read + the instructions of that script for details of the more complex + procedure. + + << This isn't really true anymore, and we should place a more + elaborate description of the cross-compiling process here >> + + When cross-compiling, there are two sorts of bootscripts that can be + used: Those that want to be executed prior to compiling and loading + the cross-compiler, which should be placed in the file called + target:cross-bootstrap.lisp, and those that should happen after the + cross-compiler has been compiled and loaded, just prior to compiling + the target, which should be placed in target:bootstrap.lisp, just + like when doing a normal recompile. + + Additionally, sometimes customized cross-compiler setup scripts + (to be used in place of e.g. cross-x86-x86.lisp) are required, + which are also placed in one of the bootfiles/*/* files. In those + cases follow the instructions provided in that file, possibly merging + the changed contents thereof with your normal cross-script. + +Step-by-Step Example of Cross-Compiling +--------------------------------------- + +This gives a step-by-step example of cross-compiling a sparc-v8 build +using a sparc-v9 build. (For some unknown reason, you can't just +remove the :sparc-v9 feature and add :sparc-v8.) + +So, first get a recent sparc-v9 build. It's best to get a version +that is up-to-date with the sources. Otherwise, you may also need to +add a bootstrap file to get any bootfiles to make your lisp +up-to-date with the current sources. + +Create a cross-compiler directory to hold the cross-compiler and a +target directory to hold the result: + + src/tools/create-target.sh xcross + src/tools/create-target.sh xtarget + +Copy the src/tools/cross-scripts/cross-sparc-sparc.lisp to +xtarget/cross.lisp. Edit it appropriately. In this case, it should +look something like: + + (c::new-backend "SPARC" + ;; Features to add here + '(:sparc :sparc-v8 + :complex-fp-vops + :linkage-table + :gencgc + :stack-checking + :relative-package-names + :conservative-float-type + :hash-new :random-mt19937 + :cmu :cmu19 :cmu19a + ) + ;; Features to remove from current *features* here + '(:sparc-v9 :sparc-v7 :x86 :x86-bootstrap :alpha :osf1 :mips + :propagate-fun-type :propagate-float-type :constrain-float-type + :openbsd :freebsd :glibc2 :linux :pentium + :long-float :new-random :small)) + + (setf *features* (remove :sparc-v9 *features*)) + (pushnew :sparc-v8 *features*) + +It's important to add frob *features* here as well as in the +new-backend. If you don't adjust *features*, they won't be set +appropriately in the result. + +Now compile the result: + + src/tools/cross-build-world.sh xtarget xcross xtarget/cross.lisp [v9 binary] + +When this finishes, you need to compile the C code: + + src/tools/rebuild-lisp.sh xtarget + +At this point, you may want to run cross-build-world.sh again to +generate a new kernel.core. It shouldn't build anything; just loads +everything and creates a kernel.core. + +With the new kernel.core, we need to create a lisp.core: + + src/tools/load-world.sh xtarget "new lisp" + +Test the result with + + xtarget/lisp/lisp -noinit + +You may now want to use this cross-compiled lisp to rebuild itself, to +make sure you haven't messed up. You can do a normal build as +described above, except your build-lisp is xtarget/lisp/lisp. + -- GitLab