Extensively reworked new implementation for specifiying jar pathnames.

Pathname namestrings that have the form "jar:URL!/ENTRY" now construct
references to the ENTRY within a jar file that is located by URL.  The
most common use is the "file:" form of URL
(e.g. 'jar:file:/home/me/foo.jar!/foo.lisp') although any valid syntax
accepted by the java.net.URL constructor should work (such as
'jar:http://abcl-dynamic-install.googlecode.com/files/baz.jar!/a/b/eek.lisp').

The internal structure of a jar pathname has changed.  Previously a
pathname with a DEVICE that was itself a pathname referenced a jar.
This convention was not able to simultaneously represent bothjar
entries that were themselves jar files as occurs with packed FASLs
within JARs and devices which refer to drive letters under Windows.
Now, a pathname which refers to a jar has a DEVICE which is a proper
list of at most two entries.  The first entry always references the
"outer jar", and the second entry (if it exists) references the "inner
jar".  Casual users are encouraged not to manipulate the "internal
structure" of jar pathname by setting its DEVICE directly, but instead
rely on namestring <--> pathname conversions.

Jar pathnames are only currently valid for use with LOAD, TRUENAME,
PROBE-FILE and pathname translation related functions (such as
MERGE-PATHNAMES, TRANSLATE-PATHNAME, etc.)  Passing one to OPEN
currently signals an error.  Jar pathnames do not currently work
with DIRECTORY or PROBE-DIRECTORY.

Jar pathnames work for ASDF systems packaged within JARs.  We override
ASDF:LOAD-OP to load ASDF from JAR Pathnames by bypassing compilation
if the output location would be in a JAR file.  Interaction with
ASDF-BINARY-LOCATIONS is currently untested.

Pathname now used as the basis of ABCL's internal routines for loading
FASLs replacing the use of strings, which simplifies a lot of the
behavior in looking for things to LOAD.

Fixed nasty shared structure bug on MERGE-PATHNAMES by implementing
(and using) a copy constructor for Pathname.

Implemented SYS:PATHNAME-JAR-P predicate for jar pathnames.

Removed ZipCache as it is no longer used now that we are using JVM's
implicit JAR caching.

WRITE-FILE-DATE works for jar pathnames, returning 0 for a
non-existent entry.

JAR-FILE tests now include loading FASLs from network location, which
means that these tests will fail if there is no network
connectivity. The tests initialization rewritten in Lisp, so it works
under Windows.

Allow of a top directory for creating hierarchially ZIPs with SYS:ZIP.
There is now a three argument version--PATHNAME PATHNAMES &OPTIONAL
TOPDIR--whereby all pathnames will be interpolated relative to topdir.

Implementation of SYS:UNZIP to unpack ZIP/JAR files.

JAR files always use '/' to name hierarchial entries. Pathname
translates '/' --> '\' under isPlatformWindows for all hierarchy
*except* reference to jar entries.

Pathname URL constructor under Windows to properly parses the
drive letter.

Ensure that *EXT:LISP-HOME* contains a directory.


Removed unused imports.

Converted Primitives to stack-trace friendly form where we touched the
source extensively anyways.
parent c8bf6ae8
JARs and JAR entries in ABCL
============================
Mark Evenson
Created: 09 JAN 2010
Modified: 24 JAN 2010
Notes towards sketching an implementation of "jar:" references to be
contained in PATHNAMEs within ABCL
Goals
-----
1. Use Common Lisp pathnames to refer to entries in a JAR file.
2. Use 'jar:' schema as documented in java.net.JarURLConnection for
namestring representation.
An entry in a JAR file:
#p"jar:file:baz.jar!/foo"
A JAR file:
#p"jar:file:baz.jar!/"
A JAR file accessible via URL
#p"jar:http://example.org/abcl.jar!/"
An entry in a ABCL FASL in a URL accessible JAR file
#p"jar:jar:http://example.org/abcl.jar!/foo.abcl!/foo-1.cls"
3. MERGE-PATHNAMES working for JAR entries
(merge-pathnames "foo-1.cls" "jar:jar:file:baz.jar!/foo.abcl!/foo._")
"jar:jar:file:baz.jar!/foo.abcl!/foo-1.cls"
(merge-pathnames "foo-1.cls" "jar:file:foo.abcl!/")
"jar:file:foo.abcl!/foo-1.cls"
4. TRUENAME and PROBE-FILE working with "jar:"
4.1 TRUENAME cannonicalizing the JAR reference.
5. DIRECTORY working within JAR files (and within JAR in JAR).
6. References "jar:<URL>" for all strings <URL> that java.net.URL can
resolve works.
Implementation
--------------
Using PATHNAMES
* A PATHNAME refering to a file within a JAR is known as a JAR
PATHNAME. It can either refer to the entire JAR file or an entry
within the JAR file.
* A JAR PATHNAME always has a DEVICE which is a proper list. This
distinguishes it from other uses of Pathname.
* The DEVICE of a JAR PATHNAME will be a list with either one or two
elements. The first element of the JAR PATHNAME can be either a
PATHNAME representing a JAR on the filesystem, or a SimpleString
representing a URL.
* a PATHNAME occuring in the list in the DEVICE of a JAR PATHNAME is
known as a DEVICE PATHNAME.
* If the DEVICE is a String it must be a String that successfully
constructs a URL via the java.net.URL(String) constructor
* Only the first entry in the the DEVICE list may be a String.
* Otherwise the the DEVICE PATHAME denotes the PATHNAME of the JAR file
* The DEVICE PATHNAME list of enclosing JARs runs from outermost to
innermost.
Use Cases
---------
// UC1 -- JAR
pathname: {
namestring: "jar:file:foo/baz.jar!/"
device: (
pathname: {
device: "jar:file:"
directory: (:RELATIVE "foo")
name: "baz"
type: "jar"
}
)
}
// UC1 -- JAR entry
pathname: {
namestring: "jar:file:baz.jar!/foo.abcl"
device: ( pathname: {
device: "jar:file:"
name: "baz"
type: "jar"
})
name: "foo"
type: "abcl"
}
// UC3 -- JAR file in a JAR entry
pathname: {
namestring: "jar:jar:file:baz.jar!/foo.abcl!/"
device: (
pathname: {
name: "baz"
type: "jar"
}
pathname: {
name: "foo"
type: "abcl"
}
)
}
// UC4 -- JAR entry in a JAR entry with directories
pathname: {
namestring: "jar:jar:file:a/baz.jar!/b/c/foo.abcl!/this/that/foo-20.cls"
device: (
pathname {
directory: (:RELATIVE "a")
name: "bar"
type: "jar"
}
pathname {
directory: (:RELATIVE "b")
name: "foo"
type: "abcl"
}
)
directory: (:RELATIVE "this" "that")
name: "foo-20"
type: "cls"
}
// UC5 -- JAR Entry in a JAR Entry
pathname: {
namestring: "jar:jar:file:a/foo/baz.jar!/foo.abcl!/a/b/bar-1.cls"
device: (
pathname: {
device: "jar:file:"
name: "baz"
type: "jar"
}
pathname: {
name: "foo"
type: "abcl"
}
)
name: "bar-1"
type: "cls"
}
// UC6 -- JAR entry in a http: accessible JAR file
pathname: {
namestring: "jar:http://example.org/abcl.jar!/org/armedbear/lisp/Version.class",
device: (
"http://example.org/abcl.jar"
pathname: {
directory: (:relative "org" "armedbear" "lisp")
name: "Version"
type: "class"
}
}
// UC7 -- JAR Entry in a JAR Entry in a URL accessible JAR FILE
pathname: {
namestring "jar:jar:http://example.org/abcl.jar!/foo.abcl!/foo-1.cls"
device: (
"http://example.org/abcl.jar"
pathname: {
name: "foo"
type: "abcl"
}
)
name: "foo-1"
type: "cls"
}
// UC8 -- JAR in an absolute directory
pathame: {
namestring: "jar:file:/a/b/foo.jar!/"
device: (
pathname: {
directory: (:ABSOLUTE "a" "b")
name: "foo"
type: "jar"
}
)
}
// UC9 -- JAR in an relative directory with entry
pathname: {
namestring: "jar:file:a/b/foo.jar!/c/d/foo.lisp"
device: (
directory: (:RELATIVE "a" "b")
name: "foo"
type: "jar"
)
directory: (:RELATIVE "c" "d")
name: "foo"
type: "lisp
}
Problems
--------
1. DEVICE PATHNAMES require the context within the nested PATHNAME
structure to be interpreted correctly.
Result: Be careful when manipulating PATHNAMEs that refer to JARs
History
-------
In the use of PATHNAMEs linked by the DEVICE field, we found the problem
that UNC path support uses the DEVICE field
Result: JARs located on UNC mounts can't be referenced. via '\\'.
i.e. jar:jar:file:\\server\share\a\b\foo.jar!/this\that!/foo.java
would not have
Solution: Instead of having DEVICE point to a PATHNAME, have DEVICE
be a list of PATHNAMES
pathname: {
namestring: "jar:jar:file:\\server\share\foo.jar!/foo.abcl!/"
device: ( pathname: {
name: "foo"
type: "abcl"
}
pathname: {
host: "server"
device: "share"
name: "foo"
type: "jar"
}
}
Which order for the list? Outermost first or last? Outermost first.
......@@ -667,6 +667,7 @@ public class Autoload extends Function
autoload(PACKAGE_SYS, "simple-list-remove-duplicates", "simple_list_remove_duplicates");
autoload(PACKAGE_SYS, "single-float-bits", "FloatFunctions", true);
autoload(PACKAGE_SYS, "std-allocate-instance", "StandardObjectFunctions", true);
autoload(PACKAGE_SYS, "unzip", "unzip", true);
autoload(PACKAGE_SYS, "zip", "zip", true);
autoload(PACKAGE_SYS, "proxy-preloaded-function",
......
......@@ -246,69 +246,72 @@ public class AutoloadedFunctionProxy extends Function {
return new JavaObject(new Hashtable());
}
// ### proxy-preloaded-function
final private static Primitive PROXY_PRELOADED_FUNCTION
= new Primitive("proxy-preloaded-function", PACKAGE_SYS, false,
"symbol name")
{
@Override
final public LispObject execute(LispObject symbol, LispObject name) {
LispThread thread = LispThread.currentThread();
Symbol sym;
Function fun;
FunctionType fType = FunctionType.NORMAL;
if (symbol instanceof Symbol)
sym = (Symbol)symbol;
else if (isValidSetfFunctionName(symbol)) {
sym = (Symbol)symbol.cadr();
fType = FunctionType.SETF;
} else if (isValidMacroFunctionName(symbol)) {
sym = (Symbol)symbol.cadr();
fType = FunctionType.MACRO;
} else {
checkSymbol(symbol); // generate an error
return null; // not reached
// ### proxy-preloaded-function symbol name => function
final private static Primitive PROXY_PRELOADED_FUNCTION = new proxy_preloaded_function();
final private static class proxy_preloaded_function extends Primitive {
proxy_preloaded_function() {
super("proxy-preloaded-function", PACKAGE_SYS, false,
"symbol name");
}
LispObject cache = AUTOLOADING_CACHE.symbolValue(thread);
if (cache instanceof Nil)
// during EVAL-WHEN :compile-toplevel, this function will
// be called without a caching environment; we'll need to
// forward to the compiled function loader
return loadCompiledFunction(name.getStringValue());
else {
LispObject[] cachedSyms = new LispObject[symsToSave.length];
for (int i = 0; i < symsToSave.length; i++)
cachedSyms[i] = symsToSave[i].symbolValue(thread);
fun = new AutoloadedFunctionProxy(sym, name, cache,
cachedSyms, fType);
fun.setClassBytes((byte[])((Hashtable)cache.javaInstance())
.get(name.getStringValue()));
@Override
final public LispObject execute(LispObject symbol, LispObject name) {
LispThread thread = LispThread.currentThread();
Symbol sym;
Function fun;
FunctionType fType = FunctionType.NORMAL;
if (symbol instanceof Symbol)
sym = (Symbol)symbol;
else if (isValidSetfFunctionName(symbol)) {
sym = (Symbol)symbol.cadr();
fType = FunctionType.SETF;
} else if (isValidMacroFunctionName(symbol)) {
sym = (Symbol)symbol.cadr();
fType = FunctionType.MACRO;
} else {
checkSymbol(symbol); // generate an error
return null; // not reached
}
LispObject cache = AUTOLOADING_CACHE.symbolValue(thread);
if (cache instanceof Nil)
// during EVAL-WHEN :compile-toplevel, this function will
// be called without a caching environment; we'll need to
// forward to the compiled function loader
return loadCompiledFunction(name.getStringValue());
else {
LispObject[] cachedSyms = new LispObject[symsToSave.length];
for (int i = 0; i < symsToSave.length; i++)
cachedSyms[i] = symsToSave[i].symbolValue(thread);
fun = new AutoloadedFunctionProxy(sym, name, cache,
cachedSyms, fType);
fun.setClassBytes((byte[])((Hashtable)cache.javaInstance())
.get(name.getStringValue()));
}
return fun;
}
return fun;
}
};
// ### function-preload
final private static Primitive FUNCTION_PRELOAD
= new Primitive("function-preload", PACKAGE_SYS, false, "name")
{
@SuppressWarnings("unchecked")
@Override
final public LispObject execute(LispObject name) {
String namestring = name.getStringValue();
LispThread thread = LispThread.currentThread();
Hashtable cache
= (Hashtable)AUTOLOADING_CACHE.symbolValue(thread).javaInstance();
byte[] bytes = readFunctionBytes(namestring);
cache.put(namestring, bytes);
return T;
}
};
// ### function-preload name => success
final private static Primitive FUNCTION_PRELOAD = new function_preload();
private static class function_preload extends Primitive {
function_preload() {
super("function-preload", PACKAGE_SYS, false, "name");
}
@SuppressWarnings("unchecked")
@Override
final public LispObject execute(LispObject name) {
String namestring = name.getStringValue();
LispThread thread = LispThread.currentThread();
Hashtable cache
= (Hashtable)AUTOLOADING_CACHE.symbolValue(thread).javaInstance();
Pathname pathname = new Pathname(namestring);
byte[] bytes = readFunctionBytes(pathname);
cache.put(namestring, bytes);
return T;
}
}
}
......@@ -280,12 +280,17 @@ public final class FileStream extends Stream
{
final Pathname pathname;
if(first instanceof Pathname) {
if (first instanceof Pathname) {
pathname = (Pathname) first;
}
else {
return type_error(first, Symbol.PATHNAME);
}
if (pathname.isJar()) {
error(new FileError("Direct stream input/output on entries in JAR files no currently supported.",
pathname));
}
final LispObject namestring = checkString(second);
LispObject elementType = third;
LispObject direction = fourth;
......
......@@ -285,8 +285,8 @@ public final class Interpreter
if (i + 1 < args.length) {
if (arg.equals("--load"))
Load.load(new Pathname(args[i + 1]),
args[i + 1],
false, false, true);
else
Load.loadSystemFile(args[i + 1]);
++i;
......
......@@ -1201,134 +1201,63 @@ public final class Lisp
LispThread.currentThread());
}
@Deprecated
public static final LispObject loadCompiledFunction(final String namestring)
{
byte[] bytes = readFunctionBytes(namestring);
Pathname name = new Pathname(namestring);
byte[] bytes = readFunctionBytes(name);
if (bytes != null)
return loadClassBytes(bytes);
return null;
}
public static final byte[] readFunctionBytes(final String namestring)
{
final LispThread thread = LispThread.currentThread();
final boolean absolute = Utilities.isFilenameAbsolute(namestring);
LispObject device = NIL;
final Pathname defaultPathname;
if (absolute)
{
defaultPathname =
coerceToPathname(Symbol.DEFAULT_PATHNAME_DEFAULTS.symbolValue(thread));
}
else
{
LispObject loadTruename = Symbol.LOAD_TRUENAME.symbolValue(thread);
if (loadTruename instanceof Pathname)
{
defaultPathname = (Pathname) loadTruename;
// We're loading a file.
device = ((Pathname)loadTruename).getDevice();
}
else
{
defaultPathname =
coerceToPathname(Symbol.DEFAULT_PATHNAME_DEFAULTS.symbolValue(thread));
}
}
if (device instanceof Pathname) { //Loading from a jar
URL url = null;
String jar = ((Pathname)device).getNamestring();
if(jar.startsWith("jar:")) {
try {
url = new URL(jar + "!/" + namestring);
} catch (MalformedURLException ex) {
Debug.trace(ex);
}
} else {
url = Lisp.class.getResource(namestring);
}
if (url != null) {
try {
InputStream input = null;
java.io.ByteArrayOutputStream baos = new java.io.ByteArrayOutputStream();
try {
input = url.openStream();
byte[] bytes = new byte[4096];
int n = 0;
while (n >= 0) {
n = input.read(bytes, 0, 4096);
if(n >= 0) {
baos.write(bytes, 0, n);
}
}
bytes = baos.toByteArray();
return bytes;
} finally {
baos.close();
if(input != null) {
input.close();
}
}
} catch (IOException e) {
Debug.trace(e);
}
}
error(new LispError("Unable to load " + namestring));
return null; // not reached
}
Pathname pathname = new Pathname(namestring);
final File file = Utilities.getFile(pathname, defaultPathname);
if (file != null && file.isFile())
{
// The .cls file exists.
try
{
byte[] bytes = readFunctionBytes(new FileInputStream(file),
(int) file.length());
// FIXME close stream!
if (bytes != null)
return bytes;
public static final byte[] readFunctionBytes(final Pathname name) {
final LispThread thread = LispThread.currentThread();
Pathname load = null;
LispObject truenameFasl = Symbol.LOAD_TRUENAME_FASL.symbolValue(thread);
LispObject truename = Symbol.LOAD_TRUENAME.symbolValue(thread);
Pathname fasl = null;
if (truenameFasl instanceof Pathname) {
load = Pathname.mergePathnames(name, (Pathname)truenameFasl, Keyword.NEWEST);
} else if (truename instanceof Pathname) {
load = Pathname.mergePathnames(name, (Pathname) truename, Keyword.NEWEST);
} else {
load = name;
}
InputStream input = load.getInputStream();
byte[] bytes = new byte[4096];
try {
if (input == null) {
Debug.trace("Pathname: " + name);
Debug.trace("LOAD_TRUENAME_FASL: " + truenameFasl);
Debug.trace("LOAD_TRUENAME: " + truename);
Debug.assertTrue(input != null);
}
catch (FileNotFoundException fnf) {
error(new LispError("Unable to load " + pathname.writeToString()
+ ": " + fnf.getMessage()));
return null; // not reached
}
return null; // not reached
}
try
{
LispObject loadTruename = Symbol.LOAD_TRUENAME.symbolValue(thread);
String zipFileName = ((Pathname)loadTruename).getNamestring();
ZipFile zipFile = ZipCache.getZip(zipFileName);
try
{
ZipEntry entry = zipFile.getEntry(namestring);
if (entry != null)
{
byte[] bytes = readFunctionBytes(zipFile.getInputStream(entry),
(int) entry.getSize());
if (bytes != null)
return bytes;
Debug.trace("Unable to load " + namestring);
error(new LispError("Unable to load " + namestring));
return null; // not reached
}
int n = 0;
java.io.ByteArrayOutputStream baos = new java.io.ByteArrayOutputStream();
try {
while (n >= 0) {
n = input.read(bytes, 0, 4096);
if (n >= 0) {
baos.write(bytes, 0, n);
}
}
} catch (IOException e) {
Debug.trace("Failed to read bytes from "
+ "'" + name.getNamestring() + "'");
return null;
}
finally
{
ZipCache.removeZip(zipFile.getName());
bytes = baos.toByteArray();
} finally {
try {
input.close();
} catch (IOException e) {
Debug.trace("Failed to close InputStream: " + e);
}
}
catch (IOException t)
{
Debug.trace(t);
}
error(new FileError("File not found: " + namestring,
new Pathname(namestring)));
return null; // not reached
return bytes;
}
public static final Function makeCompiledFunctionFromClass(Class<?> c) {
......@@ -2395,6 +2324,7 @@ public final class Lisp
Symbol.LOAD_PRINT.initializeSpecial(NIL);
Symbol.LOAD_PATHNAME.initializeSpecial(NIL);
Symbol.LOAD_TRUENAME.initializeSpecial(NIL);
Symbol.LOAD_TRUENAME_FASL.initializeSpecial(NIL);
Symbol.COMPILE_VERBOSE.initializeSpecial(T);
Symbol.COMPILE_PRINT.initializeSpecial(T);
Symbol._COMPILE_FILE_PATHNAME_.initializeSpecial(NIL);
......
This diff is collapsed.
......@@ -45,10 +45,14 @@ public final class LogicalPathname extends Pathname
private static final HashMap map = new HashMap();
public LogicalPathname()
protected LogicalPathname()
{
}
protected LogicalPathname(Pathname p) {
super(p);
}
public LogicalPathname(String host, String rest)
{
final int limit = rest.length();
......@@ -278,28 +282,31 @@ public final class LogicalPathname extends Pathname
}
// ### canonicalize-logical-host host => canonical-host
private static final Primitive CANONICALIZE_LOGICAL_HOST =
new Primitive("canonicalize-logical-host", PACKAGE_SYS, true, "host")
{
private static final Primitive CANONICALIZE_LOGICAL_HOST = new canonicalize_logical_host();
private static class canonicalize_logical_host extends Primitive {
canonicalize_logical_host() {
super("canonicalize-logical-host", PACKAGE_SYS, true, "host");
}
@Override
public LispObject execute(LispObject arg)
{
AbstractString s = checkString(arg);
if (s.length() == 0) {
// "The null string, "", is not a valid value for any
// component of a logical pathname." 19.3.2.2
return error(new LispError("Invalid logical host name: \"" +
s.getStringValue() + '"'));
}
return canonicalizeStringComponent(s);
AbstractString s = checkString(arg);
if (s.length() == 0) {
// "The null string, "", is not a valid value for any
// component of a logical pathname." 19.3.2.2
return error(new LispError("Invalid logical host name: \"" +
s.getStringValue() + '"'));
}
return canonicalizeStringComponent(s);
}
};
}
// ### %make-logical-pathname namestring => logical-pathname
private static final Primitive _MAKE_LOGICAL_PATHNAME =
new Primitive("%make-logical-pathname", PACKAGE_SYS, true, "namestring")
{
private static final Primitive _MAKE_LOGICAL_PATHNAME = new _make_logical_pathname();
private static class _make_logical_pathname extends Primitive {
_make_logical_pathname() {
super("%make-logical-pathname", PACKAGE_SYS, true, "namestring");
}
@Override
public LispObject execute(LispObject arg)
......@@ -321,5 +328,5 @@ public final class LogicalPathname extends Pathname
}
return error(new TypeError("Logical namestring does not specify a host: \"" + s + '"'));
}
};
}
}
This diff is collapsed.
......@@ -42,40 +42,36 @@ import java.net.URLDecoder;
public final class Site
{
private static final String LISP_HOME;
private static Pathname LISP_HOME;
static {
String lispHome = System.getProperty("abcl.home");
if (lispHome == null) {
URL url = Lisp.class.getResource("boot.lisp");
if (url != null) {
String protocol = url.getProtocol();
if (protocol != null && protocol.equals("file")) {
String path = url.getPath();
try {
path = URLDecoder.decode(path, "UTF-8");
}
catch (java.io.UnsupportedEncodingException uee) {
// can't happen: Java implementations are required to
// support UTF-8
}