Opening a file with a new external format fails.
Describe the bug
The test issue.25c fails in !109 (merged) for some
reason. It's related to the fact that we're also compiling the new
external format :utf16-be
, as requested when opening the file.
To Reproduce
Here is an excerpt from the test that can be run in a repl:
(let* ((in-string (concatenate 'string '(#\greek_small_letter_alpha
#\greek_small_letter_beta)))
(expected (stream:string-encode in-string :utf16-be))
(path #p"tests/tmp/issue25c.txt"))
(with-open-file (s path :direction :output :if-exists :supersede :external-format :utf16-be)
(write-string in-string s)
(force-output s)
(file-position s 0)
(with-open-file (s1 path :direction :input :element-type '(unsigned-byte 8))
(with-output-to-string (out-stream)
(ext:run-program "cat" nil
:wait t
:input s1
:output out-stream)
(let ((out-string (get-output-stream-string out-stream)))
(format t "~S ~S~%" (length out-string) (length expected))
(format t "~S ~S~%" (map 'list #'char-code out-string)
(map 'list #'char-code expected)))))))
This produces:
; Comment: $Header: src/pcl/simple-streams/external-formats/utf-16-be.lisp $
; Byte Compiling Top-Level Form:
; Compiling DEFINE-EXTERNAL-FORMAT UTF-16-BE:
; Compiling DEFINE-EXTERNAL-FORMAT UTF-16-BE:
; Compiling DEFINE-EXTERNAL-FORMAT UTF-16-BE:
; Compiling DEFINE-EXTERNAL-FORMAT UTF-16-BE:
; Byte Compiling Top-Level Form:
File-error in function LISP::FD-OPEN:
Error creating #P"tests/tmp/issue25c.txt", path does not exist.
[Condition of type KERNEL:SIMPLE-FILE-ERROR]
Restarts:
0: [CONTINUE] Return NIL.
1: [RETRY ] Retry SLIME REPL evaluation request.
2: [ABORT ] Return to SLIME's top level.
3: Return to Top-Level.
Debug (type H for help)
(LISP::FD-OPEN #P"tests/tmp/issue25c.txt" :OUTPUT :SUPERSEDE T :CREATE NIL)
Source: Error finding source:
Error in function DEBUG::GET-FILE-TOP-LEVEL-FORM: Source file no longer exists:
target:code/fd-stream.lisp.
0]
This also shows a related problem because we can't find
"target:code/fd-stream.lisp". But we should be able to because the
search-list "target:" is (#P"default:linux-4/" #P"default:src/")
and
the search-list "default:" is the root of the cmucl build tree where I
ran this test.
Expected behavior
This should work; it used to, but lately it has been a little flaky.
Now, with !109 (merged) applied, it fails very consistently.
If we do (stream::find-external-format :utf16-be)
before running the
test, everything works as expected. Thus, something about loading the
external format while opening the file is messing things up.
Desktop (please complete the following information):
- OS: Linux and OSX
- Version: Current HEAD with !109 (merged) applied.
Additional context
Let's trace some things before running the test:
* (trace open lisp::fd-open lisp::%enumerate-matches unix:unix-open
stream::string-encode)
(OPEN LISP::FD-OPEN LISP::%ENUMERATE-MATCHES UNIX:UNIX-OPEN
STRING-ENCODE)
* <test-code>
0: (OPEN #P"ext-formats:unidata.bin"
:DIRECTION :INPUT
:ELEMENT-TYPE (UNSIGNED-BYTE 8))
1: (LISP::FD-OPEN #P"ext-formats:unidata.bin" :INPUT NIL NIL NIL NIL)
2: (LISP::%ENUMERATE-MATCHES
#P"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/././ext-formats/unidata.bin"
T
T
#<Closure Over Function (FLET #:G5 (FLET #:G0 UNIX-NAMESTRING))
{610AE281}>)
3: (STRING-ENCODE
"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/././ext-formats/unidata.bin"
:UTF-8 0
NIL
NIL)
3: STRING-ENCODE returned
"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/././ext-formats/unidata.bin"
2: LISP::%ENUMERATE-MATCHES returned NIL
2: (LISP::%ENUMERATE-MATCHES
#P"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/unidata.bin"
T
T
#<Closure Over Function (FLET #:G5 (FLET #:G0 UNIX-NAMESTRING))
{610EA9E9}>)
3: (STRING-ENCODE
"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./.."
:UTF-8 0
NIL
NIL)
3: STRING-ENCODE returned
"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./.."
3: (STRING-ENCODE
"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/unidata.bin"
:UTF-8 0
NIL
NIL)
3: STRING-ENCODE returned
"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/unidata.bin"
2: LISP::%ENUMERATE-MATCHES returned
("/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/unidata.bin")
2: (UNIX:UNIX-OPEN
"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/unidata.bin"
0
438)
3: (STRING-ENCODE
"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/unidata.bin"
:UTF-8 0
NIL
NIL)
3: STRING-ENCODE returned
"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/unidata.bin"
2: UNIX:UNIX-OPEN returned 3 0
1: LISP::FD-OPEN returned
3
"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/unidata.bin"
NIL
NIL
0: OPEN returned
#<Stream for file "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/unidata.bin">
0: (STRING-ENCODE "αβ" :UTF16-BE 0 NIL NIL)
1: (OPEN "ext-formats:utf-16-be.lisp" :IF-DOES-NOT-EXIST NIL :EXTERNAL-FORMAT :ISO8859-1)
2: (LISP::FD-OPEN #P"ext-formats:utf-16-be.lisp" :INPUT NIL NIL NIL T)
3: (LISP::%ENUMERATE-MATCHES #P"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/././ext-formats/utf-16-be.lisp" T T #<Closure Over Function (FLET #:G5 (FLET #:G0 EXTENSIONS:UNIX-NAMESTRING)) {61154361}>)
4: (STRING-ENCODE "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/././ext-formats/utf-16-be.lisp" :UTF-8 0 NIL NIL)
4: STRING-ENCODE returned "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/././ext-formats/utf-16-be.lisp"
3: LISP::%ENUMERATE-MATCHES returned NIL
3: (LISP::%ENUMERATE-MATCHES #P"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/utf-16-be.lisp" T T #<Closure Over Function (FLET #:G5 (FLET #:G0 EXTENSIONS:UNIX-NAMESTRING)) {6115B929}>)
4: (STRING-ENCODE "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./.." :UTF-8 0 NIL NIL)
4: STRING-ENCODE returned "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./.."
4: (STRING-ENCODE "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/utf-16-be.lisp" :UTF-8 0 NIL NIL)
4: STRING-ENCODE returned "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/utf-16-be.lisp"
3: LISP::%ENUMERATE-MATCHES returned ("/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/utf-16-be.lisp")
3: (UNIX:UNIX-OPEN "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/utf-16-be.lisp" 0 438)
4: (STRING-ENCODE "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/utf-16-be.lisp" :UTF-8 0 NIL NIL)
4: STRING-ENCODE returned "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/utf-16-be.lisp"
3: UNIX:UNIX-OPEN returned 3 0
2: LISP::FD-OPEN returned 3 "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/utf-16-be.lisp" NIL NIL
1: OPEN returned #<Stream for file "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/utf-16-be.lisp">
1: (LISP::%ENUMERATE-MATCHES #P"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/././ext-formats/" T T #<Closure Over Function (FLET #:G5 (FLET #:G0 EXTENSIONS:UNIX-NAMESTRING)) {61179611}>)
2: (STRING-ENCODE "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/././ext-formats/" :UTF-8 0 NIL NIL)
2: STRING-ENCODE returned "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/././ext-formats/"
1: LISP::%ENUMERATE-MATCHES returned NIL
1: (LISP::%ENUMERATE-MATCHES #P"/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/" T T #<Closure Over Function (FLET #:G5 (FLET #:G0 EXTENSIONS:UNIX-NAMESTRING)) {611812C1}>)
2: (STRING-ENCODE "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./.." :UTF-8 0 NIL NIL)
2: STRING-ENCODE returned "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./.."
2: (STRING-ENCODE "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/" :UTF-8 0 NIL NIL)
2: STRING-ENCODE returned "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/"
1: LISP::%ENUMERATE-MATCHES returned ("/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/")
1: (STRING-ENCODE "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/" :UTF-8 0 NIL NIL)
1: STRING-ENCODE returned "/home/toy/src/clnet/cmucl/dev/cmucl/dist-linux/bin/./../lib/cmucl/lib/ext-formats/"
1: (UNIX:UNIX-OPEN "/dev/null" 0 438)
1: (LISP::%ENUMERATE-MATCHES #P"/home/toy/src/clnet/cmucl/dev/cmucl/" T T #<Closure Over Function (FLET #:G5 (FLET #:G0 EXTENSIONS:UNIX-NAMESTRING)) {61199871}>)
2: (STRING-ENCODE "/home/toy/src/clnet/cmucl/dev/cmucl/" :UTF-8 0 NIL NIL)
2: STRING-ENCODE returned "/home/toy/src/clnet/cmucl/dev/cmucl/"
1: LISP::%ENUMERATE-MATCHES returned ("/home/toy/src/clnet/cmucl/dev/cmucl/")
1: (STRING-ENCODE "/home/toy/src/clnet/cmucl/dev/cmucl/" :UTF-8 0 NIL NIL)
1: STRING-ENCODE returned "/home/toy/src/clnet/cmucl/dev/cmucl/"
; Comment: $Header: src/pcl/simple-streams/external-formats/utf-16-be.lisp $
; Byte Compiling Top-Level Form:
; Compiling DEFINE-EXTERNAL-FORMAT UTF-16-BE:
; Compiling DEFINE-EXTERNAL-FORMAT UTF-16-BE:
; Compiling DEFINE-EXTERNAL-FORMAT UTF-16-BE:
; Compiling DEFINE-EXTERNAL-FORMAT UTF-16-BE:
; Byte Compiling Top-Level Form:
0: STRING-ENCODE returned "±²"
0: (OPEN #P"tests/tmp/issue25c.txt"
:DIRECTION :OUTPUT
:IF-EXISTS :SUPERSEDE
:EXTERNAL-FORMAT :UTF16-BE)
1: (LISP::FD-OPEN #P"tests/tmp/issue25c.txt" :OUTPUT :SUPERSEDE T NIL NIL)
2: (LISP::%ENUMERATE-MATCHES #P"tests/tmp/issue25c.txt"
NIL
T
#<Closure Over Function (FLET #:G5
(FLET #:G0
UNIX-NAMESTRING))
{603376F1}>)
2: LISP::%ENUMERATE-MATCHES returned ("tests/tmp/issue25c.txt")
2: (UNIX:UNIX-OPEN "tests/tmp/issue25c.txt" 577 438)
3: (STRING-ENCODE "tests/tmp/issue25c.txt" :UTF-8 0 NIL NIL)
3: STRING-ENCODE returned "\0t\0e\0s\0t\0s\0/\0t\0m\0p\0/\0i\0s\0s\0u\0e\02\05\0c\0.\0t\0x\0t"
2: UNIX:UNIX-OPEN returned NIL 2
File-error in function LISP::FD-OPEN:
Error creating #P"tests/tmp/issue25c.txt", path does not exist.
[Condition of type KERNEL:SIMPLE-FILE-ERROR]
Restarts:
0: [CONTINUE] Return NIL.
1: [RETRY ] Retry SLIME REPL evaluation request.
2: [ABORT ] Return to SLIME's top level.
3: Return to Top-Level.
Debug (type H for help)
(LISP::FD-OPEN #P"tests/tmp/issue25c.txt" :OUTPUT :SUPERSEDE T :CREATE NIL)
Source:
2: (LISP::%ENUMERATE-MATCHES
#P"/home/toy/src/clnet/cmucl/dev/cmucl/linux-4/code/fd-stream.lisp"
T
T
#<Closure Over Function (FLET #:G5 (FLET #:G0 UNIX-NAMESTRING))
{60427951}>)
3: (STRING-ENCODE
"/home/toy/src/clnet/cmucl/dev/cmucl/linux-4/code/fd-stream.lisp"
:UTF-8 0
NIL
NIL)
3: STRING-ENCODE returned
"\0/\0h\0o\0m\0e\0/\0t\0o\0y\0/\0s\0r\0c\0/\0c\0l\0n\0e\0t\0/\0c\0m\0u\0c\0l\0/\0d\0e\0v\0/\0c\0m\0u\0c\0l\0/\0l\0i\0n\0u\0x\0-\04\0/\0c\0o\0d\0e\0/\0f\0d\0-\0s\0t\0r\0e\0a\0m\0.\0l\0i\0s\0p"
2: LISP::%ENUMERATE-MATCHES returned NIL
2: (LISP::%ENUMERATE-MATCHES
#P"/home/toy/src/clnet/cmucl/dev/cmucl/src/code/fd-stream.lisp"
T
T
#<Closure Over Function (FLET #:G5 (FLET #:G0 UNIX-NAMESTRING))
{604381E9}>)
3: (STRING-ENCODE
"/home/toy/src/clnet/cmucl/dev/cmucl/src/code/fd-stream.lisp"
:UTF-8 0
NIL
NIL)
3: STRING-ENCODE returned
"\0/\0h\0o\0m\0e\0/\0t\0o\0y\0/\0s\0r\0c\0/\0c\0l\0n\0e\0t\0/\0c\0m\0u\0c\0l\0/\0d\0e\0v\0/\0c\0m\0u\0c\0l\0/\0s\0r\0c\0/\0c\0o\0d\0e\0/\0f\0d\0-\0s\0t\0r\0e\0a\0m\0.\0l\0i\0s\0p"
2: LISP::%ENUMERATE-MATCHES returned NIL
Error finding source:
Error in function DEBUG::GET-FILE-TOP-LEVEL-FORM: Source file no longer exists:
target:code/fd-stream.lisp.
0]
We can see that there's some kind of confusion here. When opening
"unidata.bin", string-encode
returns a string that appears identical
to the input. The same thing happens when we open "utf-16-be.lisp".
But when we try opening (string-encode "tests/tmp/issue25c.txt")
returns
"\0t\0e\0s\0t\0s\0/\0t\0m\0p\0/\0i\0s\0s\0u\0e\02\05\0c\0.\0t\0x\0t".
And now we can't find the file.
But if we run the test again, after the utf-16-be format has been loaded, we see (in part):
0: (OPEN #P"tests/tmp/issue25c.txt"
:DIRECTION :INPUT
:ELEMENT-TYPE (UNSIGNED-BYTE 8))
1: (LISP::FD-OPEN #P"tests/tmp/issue25c.txt" :INPUT NIL NIL NIL NIL)
2: (LISP::%ENUMERATE-MATCHES #P"tests/tmp/issue25c.txt"
T
T
#<Closure Over Function (FLET #:G5
(FLET #:G0
UNIX-NAMESTRING))
{61388CF9}>)
3: (STRING-ENCODE "tests/tmp/issue25c.txt" :UTF-8 0 NIL NIL)
3: STRING-ENCODE returned "tests/tmp/issue25c.txt"
2: LISP::%ENUMERATE-MATCHES returned ("tests/tmp/issue25c.txt")
2: (UNIX:UNIX-OPEN "tests/tmp/issue25c.txt" 0 438)
3: (STRING-ENCODE "tests/tmp/issue25c.txt" :UTF-8 0 NIL NIL)
3: STRING-ENCODE returned "tests/tmp/issue25c.txt"
2: UNIX:UNIX-OPEN returned 5 0
1: LISP::FD-OPEN returned 5 "tests/tmp/issue25c.txt" NIL NIL
0: OPEN returned #<Stream for file "tests/tmp/issue25c.txt">
So now string-encode
doesn't have the embedded 0s.
Hmm. it seems that in the first case string-encode
was actually
encoding with utf-16-be:
* (string-encode "tests/tmp/issue25c.txt" :utf-16-be)
"\0t\0e\0s\0t\0s\0/\0t\0m\0p\0/\0i\0s\0s\0u\0e\02\05\0c\0.\0t\0x\0t"
How is that possible? In the broken test, we explicitly say we're
using :utf-8
for the encoding.