Broadly speaking, add version numbers to files and directories in the CMUCL distribution.
The purpose of this addition is to allow multiple distributions of CMUCL unpacked into the same location, like /usr/local.
This is basically what Emacs, Octave, Python, etc. do to ensure multiple versions of those packages can coexist in the same directory hierarchy.
Is there a prototype?
None, yet.
Describe the feature in more detail
I think this basically entails the following
Rename files in shared locations like bin and man appending a version number and provide a symlink from the old name to the new name. For example, we would rename bin/lisp to bin/lisp-21e and provide a symlink from bin/lisp to bin/lisp-21e.
Add a version number to the library directory and teach the logic for searching for the library directory to use the version number. For example, instead of lib/cmucl/lib we would use lib/cmucl-21e/lib or lib/cmucl/21e/lib This will require threading a version string into the build that we can add to cmucllib_search_list in lisp/lisp.c among other locations
Designs
Child items
...
Show closed items
Linked items
0
Link issues together to show that they're related.
Learn more.
For releases, the version number is pretty obvious. However, what about snapshots? For my own use, I've been putting snapshot-yyyy-mm in the directory yyyy-mm. But I also apparently use the full snapshot name.
So are you expecting that I could untar the releases and snapshots all into the same root directory and keep all the different versions nicely separated? So bin would have bin/lisp-21e, bin/lisp-2024-04, etc.?
If we did this consistently, that would make it fairly easy to figure
out the core file and other paths. We could look at the executable
name and extract everything after "lisp-" to get the version number to
use. Then we could look in lib/cmucl-nnn as the directory containing
everything else for the associated files for that version.
(Kind of forgotten how easy or hard it is to get the executable name
and how links might change that.)
There are a few questions here, I think this answers each of them
The naming convention(s) can be whatever we want. For snapshots, a date with the month and year, like yyyy-mm, is probably sufficient. I'm not sure what the full snapshot name. If you can give me an example that might help identify any contradictions
Yes! The idea is that you can untar the releases and snapshots into the same directory and have everything from commingling. The only exception will be symlinks to binaries and man pages.
The version will be baked into the binary (21e, 2024-12, etc.) so we don't have to use the binary name which, as you suggest, can be tricky.
Currently, a snapshot is built by creating a tag snapshot-yyyy-mm. The tarballs are named cmucl-yyyy-mm-.... Similarly, a release is done by creating a tag 21e. But if you create a tarball from some random branch (or master), you get something like snapshot-yyyy-mm-g<hash>-dirty. The tarballs have the name yyyy-mm-g<hash>-dirty. The version comes from git describe --dirty.
Actually, we use bin/make-dist.sh to create the tarballs. In the process of making them, we copy the appropriate files to a new directory which we tar up. Thus, when we copy the files over, we can rename them as needed. And make-dist.sh takes a -V option to specify the version. The default is taken from git describe but we strip off the "snapshot" part if any. This probably takes care of the naming of the files.
What remains is having lisp determine what version is being used. Perhaps what we can do is add a generated C file (like we have version.c) that contains a string with the version. This gets compiled in.
I think, though, we might need two paths. One that has the version info and one that serves as the default. Why? We need the different builds to be able to find the core file that was just created so we can continue the build. Or maybe we need to make a dist each time before starting the next build. That also means cross-compiles need to do the same. This slows building a bit, but just a few seconds or so, on my laptop.
What remains is having lisp determine what version is being used. Perhaps what we can do is add a generated C file (like we have version.c) that contains a string with the version. This gets compiled in.
A macro should be define. By default it can be
#ifndef VERSION#define VERSION "alpha"#endif
And then we just pass -DVERSION=21e to the compiler when doing a build for a named version.
I think, though, we might need two paths. One that has the version info and one that serves as the default. Why? We need the different builds to be able to find the core file that was just created so we can continue the build.
Like the stage2 build finding the stage1 lisp.core? The version info is only interesting if the user does specify the -core flag with a path to the core file. I would hope that the build uses that already. Does it not?
If someone does not specify a core file than, yeah, we want to use the version information to sniff out the location of the core file.
Or maybe we need to make a dist each time before starting the next build. That also means cross-compiles need to do the same. This slows building a bit, but just a few seconds or so, on my laptop.
If we're relying on finding the core file automatically during a build, can you show me where that is happening in the build scripts? Maybe this is easy to fix and engineer away the problem you are describing.
I think a macro will work fine, but we'll also need to create a string or something so we can find the corresponding lib directory that contains the clm, hemlock, clx, and contrib stuff. This is needed to set up the "library:" search-list.
If we use the -core command-line arg, we don't have to care about each stage getting the right core file version.
Let's make this a little more concrete. Currently, if you extract the two tarballs, you get a directory structure:
bin/ lisp <lots of script files for building, running tests, etc.>doc/ cmucl/README (we should probably get rid of this?)lib/ cmucl/ internals.h internals.inc lisp.map lisp.nm sample-wrapper lib/ <files for creating an executable> <lisp core> motifd <other scripts> <hemlock stuff> contrib/ subsystems/ clx, clm, gray-streams, hemlock, simple-streams locale/man/ man1/src/tests/
I guess the proposal here is the directory structure now looks like:
bin/ lisp-$versiondoc/ cmucl/README (we should probably get rid of this?)lib/ cmucl-$version/ <same contents as before>man/ man1/src/tests/
Not sure what to name the the man, src, and tests directories. If we
update "target:" search-list appropriately (in the default site-init
file), the name of the src directory won't really matter.
We need to update the scripts in bin to know the name of tests
directory.
Also, the bin directory has lots of scripts. I suspect most of these
aren't really relevant unless you're a developer. Maybe put them in a
subdirectory named tools-$version? Perhaps the only files a user
might run would be the scripts to run the ansi and unit tests.
bin/lisp -> bin/lisp-$version (symlink)bin/lisp-$versionlib/cmucl/$version/...share/man/man1/lisp.1 -> man/man1/lisp-$version.1 (symlink)share/man/man1/lisp-$version.1share/cmucl/$version/doc/...share/cmucl/$version/src/...etc. etc. etc.
I am not exactly sure about the tests, they probably belong somewhere under share/cmucl/$version/ as well.
None of this should be especially novel, it is basically the prevailing convention for packaging UNIX software. On that note, the lib directory is a bit of a dumping ground and eventually should have some of its contents moved elsewhere. For example, motifd probably belongs under libexec/cmucl/$version.
I think that's about it. I have a very rough prototype in !261 (merged). Needs lots of cleanup, but I'm just trying to get things to kind of work. It's nice we have a -lib command-line option to specify the library search list so we can easily find core file when running build.sh.
The packaging scripts handle most of the work now. Packaging the src code isn't done yet.
Grab or browse the
artifacts
to see what the current versioning looks like. The good news is that
CI is happy with these changes so far, even if they are really ugly.
Still haven't decided where to put all the scripts in bin. Plus, a
user very likely won't be able to use them to build lisp or run tests
as they are currently set up. We could probably fix this, but it's a
low priority for me right now.
Yep. I forgot about that until I after I started using the -lib option. Not sure which is better, but at least -lib is explicit. There's also the -core option, which could work, except we'd need to know the default core on x86 is lisp-sse2.core, not lisp.core. Now that we only do sse2, we could rename it lisp.core again. Could remove a bunch of code as well.
Oh, it looks like we don't actually distribute the other shell scripts in bin. And we also don't distribute the tests in directory tests. At least we didn't in the 2024-08 snapshot.
That makes things a bit easier. But should we distribute the tests and the scripts to run them? This would mean the scripts need to know where the test files are.
Yes. Probably an oversight from long ago where we forgot to include the tests. That means we also need to include the scripts to run the tests. Well, at least the unit tests. The ansi tests require the user to get the tests themselves.
We'd also need a way to tell the user where the tests are. I'd prefer not to embed the version info in the script.
But what about the other scripts, like clean-target.sh, build.sh, rebuild-lisp.sh and such for building cmucl? We don't include those either.