Split UIOP into separate project?
Apologies for the long issue, but I'm proposing a seismic shift in how ASDF has been developed since v3. So I think a lot of justification and discussion is warranted.
Because of ASDF's dependence on UIOP (which itself uses ASDF), they have a very special relationship. This works well enough most of the time, but does cause some difficulties here and there.
There's special casing for UIOP in several ASDF files (see at least: 1, 2 (and others in that file), and 3). Special cases always add some maintainer overhead and this could cause issues when new features are introduced if a special case needed for UIOP is missed. There's been at least one issue opened where this special relationship has caused some difficulty doing something that should be possible. See: #12.
Most importantly to me, this relationship can make it difficult on CL dependency managers in a variety of ways.
First, while having UIOP always available if ASDF is loaded is great for adoption, it encourages (IMHO) bad practice in declaring dependencies. How many systems use UIOP (including uiop:define-package
) without explicitly declaring it as a dependency? Many. This actually is causing a problem in QL with ASDF/UIOP 3.3.5. See 4 / #84 (closed). Long story short, the package nickname support causes issues if the uiop:define-package
macro is expanded with v3.3.5, but the resulting fasl is then loaded in an image that has <3.3.5 loaded. This isn't really ASDF's problem, but the choice to bundle ASDF and UIOP together and the upgrade hesitancy from certain CL implementations does make it much more likely to happen.
Second, ASDF's refusal to allow UIOP to be downgraded means that dependency managers that try to pin versions of dependencies (like CLPM and Qlot) either have to special case UIOP themselves or just accept the fact that an older pinned version of UIOP will be straight up ignored if ASDF is upgraded. Being able to load older versions is important to some people because sometimes things just break and it's nice to be able to use old versions until a fix is released.
There's some good reading on the reasoning behind how this special relationship came to be at https://fare.livejournal.com/190738.html, along with some thoughts on ways forward. But I think there might be a simpler way forward: split UIOP and ASDF into separate projects, but still include UIOP as part of ASDF using mangled package names. UIOP can be built with a subclass of monolithic-concatenate-source-op
that rewrites define-package
and in-package
forms so that a package like uiop/filesystem
would become asdf/uiop/filesystem
. The result of that could then be included in the ASDF distribution, resulting in ASDF essentially having a private copy of UIOP.
Pros of this approach:
- UIOP and ASDF can be developed at different paces. This is particularly nice as more systems use UIOP directly as it allows UIOP bugs to be fixed and released without a corresponding ASDF version bump (unless they also impact ASDF directly).
- UIOP becomes just another system. All of the special casing for UIOP disappears from ASDF and CL dependency mangers.
- I'm assuming that UIOP would remain under the umbrella of ASDF developers. This gives us a great platform to demonstrate some ASDF features (like the proposed concatenation operation) actually using ASDF (as compared to the
build/asdf.lisp
target in the Makefile which usescat
).
Cons:
- Everything that assumes presence of ASDF means UIOP is also present would break. This could be mitigated in the short term by adding a
uiop
nickname to theasdf/uiop
package. But we would want to make sure that we have a clear deprecation plan and that deprecation plan also works with upgrades of ASDF and UIOP. It honestly might be easiest to give the split out UIOP a new name (a shame, since UIOP is a great name; although package local nicknames can help make that more palatable to users) and encourage everyone to use that system and the packages it defines instead of theuiop
package. - It would be difficult for a user to upgrade the version of UIOP that ASDF uses. As described, the user would have to rebuild asdf.lisp with the new UIOP. It would be much more difficult to switch UIOP implementations of an already loaded ASDF. IMO, that wouldn't be necessary to support in the short term, but I think there may be several ways of eventually supporting it.