- Jun 29, 2011
-
-
rtoy authored
stream.lisp: o Remove the old code that was the first attempt at fixing issue #43. This was wrong. fd-stream.lisp: o Fix issue #43. Even if we have a string-buffer, we need to take into account any octets in the in-buffer (if it exists) that have not been processed. This happens if the in-buffer does not have enough octets at the end to form a complete character for the given encoding. o Some debugging prints added, but disabled.
-
- May 31, 2011
-
-
rtoy authored
Comment from Paul Foley: Problem: FD-OPEN calls NEXT-VERSION with the namestring; NEXT-VERSION calls EXTRACT-NAME-TYPE-AND-VERSION, which builds a "pattern" when it sees globbing characters in the name, and NEXT-VERSION promptly crashes when it tries to concatenate strings. Fix: Bind *IGNORE-WILDCARDS* to T in NEXT-VERSION (in fd-stream.lisp)
-
- Feb 17, 2011
-
-
rtoy authored
(OPEN PN :DIRECTION :IO :IF-DOES-NOT-EXIST NIL) no longer signals an error. code/fd-stream.lisp: o Don't try to create a new version if there's no existing file general-info/release-20c.txt: o Update.
-
- Oct 13, 2010
-
-
rtoy authored
compiled with and without Unicode. This is needed so that the pot files have the same content for both unicode and non-unicode builds. (The _"" and _N"" are handled by the reader, so things that are conditionalized out still get processed, unlike using gettext.)
-
- Oct 12, 2010
-
-
rtoy authored
of streams. This is a change from current 20a and 20b behavior which didn't signal errors, but matches the behavior for releases 19f and earlier. But allow them to work on binary-text-streams. This is the same behavior as before for binary-text-stream streams. However, read-sequence no longer allows reading from streams into arbitrary objects, unless the stream is a binary-text-stream stream. code/fd-stream-extfmt.lisp: o In %SET-FD-STREAM-EXTERNAL-FORMAT, only update fd-stream-in/fd-stream-out if we have a character or binary-text-stream stream. o Don't update the fd-stream-string-buffer or lisp-stream-in-buffer if we have a binary-text-stream because that will mess up how fast-read-char and fast-read-byte dispatch to do the right thing for binary-text-stream streams. code/fd-stream.lisp: o Set the fd-stream-in and fd-stream-bin slots appropriately depending on whether we have a character, binary, or binary-text-stream stream. o Only create the lisp-stream-in-buffer if we do NOT have a binary-text-stream. (Binary streams didn't use the lisp-stream-buffer previously, so no change there. Character streams use the lisp-stream-buffer and/or lisp-string-buffer.) o Set the fd-stream-flags appropriately for the kind of stream this is. Checking a fixnum is faster than checking the type of a stream. code/struct.lisp: o Add FLAGS slot to LISP-STREAM so we can tell what kind of stream (character, binary, binary-text-stream) we have. code/sysmacs.lisp: o Change FAST-READ-CHAR so that if we have a have a binary or binary-text-stream stream, we dispatch to the fast-read-char methods to do the right thing, including signaling an error for the wrong kind of stream. o Change FAST-READ-BYTE so that if we do not have a binary stream, we dispatch to the fast-read-char method to do the right thing. compiler/dump.lisp: o With the above changes, we can no longer write characters to a binary stream, like a FASL file. Make the fasl file a binary-text-stream so that we can. (Alternatively, we could create the FASL header as a string, convert to octets and dump the octest to the file. This is easier, and should still be fast for writing fasls.)
-
- Sep 15, 2010
-
-
rtoy authored
streams with a string-buffer (all external formats except ISO-8859-1) and for streams without a string-buffer but with an in-buffer (ISO-8859-1). This caused source location information to be totally wrong.
-
- Sep 08, 2010
-
-
rtoy authored
requires a cross-compile using the default cross-compile script. fd-stream.lisp: o Remove the SAVED-OC-STATE slot from an FD-STREAM because we don't need it anymore since we have the OCTET-COUNT slot. stream.lisp: o Don't need to copy the current OC-STATE to SAVED-OC-STATE. Remove code and update comments. fd-stream-extfmt.lisp: o Use the OCTET-COUNT slot to figure out how many octets have been consumed so far to produce the characters that have already been read. Don't need to do the re-conversion anymore, so we don't need the SAVED-OC-STATE anymore. o Add support for the case where we were using the ISO8859-1 external format and are now switching to another external format that requires the string-buffer.
-
- Aug 18, 2010
-
-
rtoy authored
string-buffer-len because the string buffer has an extra element at the beginning (for unread-char).
-
- Aug 15, 2010
-
-
rtoy authored
instead of the (invalid) Unicode replacement character.
-
rtoy authored
the octets consumed for each character. This array is used to figure out the file position. Some tests comparing this scheme indicates a very small slowdown of about 1%, so this seems not to hurt. Use a cross-compile using the 2010-07 snapshot to build this. (Same procedure as used to build the 20b-pre1 release.) struct.lisp: o Add new slot OCTET-COUNT to LISP-STREAM to hold the array of octets per character. extfmts.lisp: o Add OCTETS-TO-STRING-COUNTED, which is like OCTETS-TO-STRING, except we need an array in which to store the number of octets consumed for each character processed. fd-stream.lisp: o Create the octet-count array creating the lisp stream string buffer. o In FD-STREAM-FILE-POSITION, use the octet count to count the number of octets that have been read but not yet returned to the user. stream.lisp: o Use OCTETS-TO-STRING-COUNTED instead of OCTETS-TO-STRING so we keep track of octet length of each character processed.
-
- Aug 04, 2010
-
-
rtoy authored
:DECODING-ERROR arguments.
-
- Jul 20, 2010
-
-
rtoy authored
A normal build can be used, but you'll have to select the clobber-it restart each time you're asked during the normal build.
-
- Jul 19, 2010
-
-
rtoy authored
reference fd-stream-char-to-octets-error and don't call %make-fd-stream with :char-to-octets-error and :octets-to-char-error since these don't exist without unicode.
-
- Jul 05, 2010
- Jul 03, 2010
-
-
rtoy authored
o Update comments for the various slots in DEFINE-EXTERNAL-FORMAT. fd-stream.lisp: o Declare the types for the CHAR-TO-OCTETS-ERROR and OCTETS-TO-CHAR-ERROR slots in FD-STREAM. o Update docstrings for MAKE-FD-STREAM and OPEN for :DECODING-ERROR and :ENCODING-ERROR parameters.
-
rtoy authored
variable that was shadowing another.
-
rtoy authored
where the external format does not have a flush-state method.
-
rtoy authored
o Add error parameter to flush-state in external format definition so we can handle errors when flushing the state to a stream. o Add optional error parameter to flush-state macro. code/fd-stream.lisp: o For the case where an external format has flush method, EF-FLUSH was not calling it correctly. Update so the output function actuall works. o Add error handler to call to flush-state. o For the case where an external format does not have a flush method, output the state value instead of a replacement character so the external format can handle any errors.
-
- Jul 02, 2010
-
-
rtoy authored
o Pass the error handler on for composed external formats. code/fd-stream.lisp: o Forgot to pass the error-handler to char-to-octets in EF-COUT. o In MAKE-FD-STREAM slightly change handling of encoding-error and decoding-error: - If :encoding-error is a character, then the external format will use that character whenever an encoding error happens instead of the default (internally specified by the external format). - If :decoding-error is a character, then the external format will use that character whenever an encoding error happens instead of the default (internally specified by the external format). If :decoding-error is T, then a cerror is signaled; if continued, the Unicode replacement character (#\U+FFFD) is used. o Fix bug in OPEN: The :decoding-error and :encoding-error keyword parameter was placed in the &aux section by mistake.
-
rtoy authored
o The optional error parameter doesn't need to be optional in DEFINE-EXTERNAL-FORMAT, EF-STRING-TO-OCTETS, EF-OCTETS-TO-STRING, EF-ENCODE and EF-DECODE. code/fd-stream.lisp: o Update comments for char-to-octets-error and octets-to-char-error. o Forgot to pass the error handler to char-to-octets in EF-SOUT and EF-STRLEN.
-
rtoy authored
-
rtoy authored
code/extfmts.lisp o Call the error handler for iso8859-1 output. o In OCTETS-TO-CODEPOINT and CODEPOINT-TO-OCTETS, call the external format with the error argument. o In OCTETS-TO-CHAR - Call OCTETS-TO-CODEPOINT with the error handler. - For all of the error conditions, call the error handler if defined. o Add error parameter to EF-STRING-TO-OCTETS and EF-ENCODE so we can handle errors. Call CHAR-TO-OCTETS with the error handler. o Add error parameter to STRING-TO-OCTETS and use it. o Add error parameter to EF-OCTETS-TO-STRING and EF-DECODE so we can handle errors. Call OCTETS-TO-CHAR with the error handler. o Add error parameter to OCTETS-TO-STRING and use it. o In STRING-ENCODE and STRING-DECODE, call the ef function with the error handler. o Change STRING-ENCODE to use keyword args instead of optional args. Add error parameter and use it. code/fd-stream-extfmt.lisp: o Tell OCTETS-TO-STRING about the error handler stored in the fd-stream. code/fd-stream.lisp: o OPEN, MAKE-FD-STREAM, and OPEN-FD-STREAM get DECODING-ERROR and ENCODING-ERROR keyword arguments for specifying how to handle decoding and encoding errors in external formats. code/stream.lisp: o Make sure the error handler is called in FAST-READ-CHAR-STRING-REFILL. pcl/simple-streams/external-formats/utf-8.lisp: o Initial cut at calling the error handler for the various possible invalid octet streams for a utf-8 encoding.
-
- Jun 30, 2010
-
-
rtoy authored
when converting octets to characters or characters to octets. These aren't currently being used, but will be. A basic cross-compile is needed. When CMUCL asks about incompatible structures, select the CLOBBER-IT restart in all cases.
-
- Jun 07, 2010
-
-
rtoy authored
o OPEN merges the filename with *DEFAULT-PATHNAME-DEFAULTS*, as required by CLHS sec 19.2.3. code/filesys.lisp: o DELETE-FILE, ENSURE-DIRECTORIES-EXIST, and FILE-AUTHOR merge the filename with *DEFAULT-PATHNAME-DEFAULTS*, as required by CLHS sec 19.2.3. general-info/release-20b.txt: o Update.
-
- Apr 20, 2010
-
-
rtoy authored
may get confused with source locations if the reader macros are installed.
-
- Apr 19, 2010
-
-
rtoy authored
-
- Mar 19, 2010
-
-
rtoy authored
boot-2010-02-1 as the bootstrap file. You should probably also use the new -P option for build.sh to generate and update the po files while building.
-
- Jan 25, 2010
-
-
rtoy authored
up another string.
-
- Jan 23, 2010
-
-
rtoy authored
everything is defined yet. Hence, add dummy %SET-FD-STREAM-EXTERNAL-FORMAT and move the real one to fd-stream-extfmt.lisp. This builds. code/fd-stream.lisp: o Always call %SET-FD-STREAM-EXTERNAL-FORMAT, even if LISP::*ENABLE-STREAM-BUFFER-P* is NIL. code/stream.lisp: o Move %SET-FD-STREAM-EXTERNAL-FORMAT to fd-stream-extfmt.lisp. o Add dummy implementation of %SET-FD-STREAM-EXTERNAL-FORMAT. code/fd-stream-extfmt.lisp: o %SET-FD-STREAM-EXTERNAL-FORMAT moved here.
-
rtoy authored
set-routines so that the stream has the specified external format.
-
- Jan 22, 2010
-
-
rtoy authored
file position, we need to convert what's left in the buffer into octets to figure out how many octets haven't been delivered yet.
-
rtoy authored
inadvertently broken when the fast external format work was done. The position did not take into account the number of octets that had been read in but not yet processed.
-
- Oct 18, 2009
-
-
rtoy authored
faster reads on external-formats. This is done by adding an additional buffer to streams so we can convert the entire in-buffer into characters all at once. To build this change, you need to do a cross-compile using boot-2009-10-1-cross.lisp. Using that build, do a normal build with these sources. For a non-unicode build use boot-2009-10-01.lisp with a 20a non-unicode build. code/extfmts.lisp: o Add another slot to the extfmts for copying the state. o Modify EF-OCTETS-TO-STRING and OCTETS-TO-STRING to support the necesssary changes for fast formats. This is incompatible with the previous version because the string is not grown if needed. code/fd-stream-extfmt.lisp: o Set *enable-stream-buffer-p* to T so we have fast streams. code/fd-stream.lisp: o Add new slots to support fast strams. o In SET-ROUTINES, initialize the new slots appropriately. o Update UNREAD-CHAR to be able to back up in the string buffer to unread. o Add implementation to copy the state of an external format. code/stream.lisp: o Change %SET-FD-STREAM-EXTERNAL-FORMAT to be able to change formats even if we've already converted the buffer with a different format. We reconvert the buffer with the old format until we reach the current character. Then the remaining octets are converted using the new format and stored in the string buffer. o Add FAST-READ-CHAR-STRING-REFILL to refill the string buffer, like FAST-READ-CHAR-REFILL does for the octet in-buffer. code/struct.lisp: o Add new slots to hold the string buffer, the current index, and length. These are needed for the fast formats. code/sysmacs.lisp: o Update PREPARE-FOR-FAST-READ-CHAR, DONE-WITH-FAST-READ-CHAR, and FAST-READ-CHAR to support the string buffer. code/string.lisp: o Microoptimization of SURROGATEP to reduce the number of branchs. general-info/release-20b.txt: o Update with these changes pcl/simple-streams/external-formats/utf-16-be.lisp: pcl/simple-streams/external-formats/utf-16-le.lisp: pcl/simple-streams/external-formats/utf-16.lisp: o These formats actually have state, so update them to take a handle an initial state. These are needed if the string buffer ends with a leading surrogate and the next string buffer starts with a trailing surrogate. The conversion needs to combine the surrogates together.
-
- Sep 09, 2009
-
-
rtoy authored
-
- Aug 26, 2009
-
-
rtoy authored
closing an output stream. This causes things like (with-open-file (s "foo" :direction :output :external-format :utf-8) (write-char #\u+d800 s)) to output the replacement character instead of creating an empty file. code/extfmts.lisp: o Add new slot to efx structure to hold the function to flush the state in an external format. o Add accessor for the flush-state slot. o Update DEFINE-EXTERNAL-FORMAT to allow specifying the flush function. o Add macro to call the flush-state function. o Added +EF-FLUSH+ o Use vm::defenum to name the constants instead of the hand-written values. o Export +REPLACEMENT-CHARACTER-CODE+ o Document the slots in an efx stucture. code/fd-stream.lisp: o Add ef-flush def-ef-macro to flush the state of an external format when closing an output file. If ef-flush-state is NIL, we just call EF-COUT to send out the replacement character. Otherwise, the flush-state function is called to handle it. o When closing an output character stream, call ef-flush to flush any state before flushing the buffers of the stream. o Document the unicode slots in an fd-stream. code/exports.lisp: o Export +REPLACEMENT-CHARACTER-CODE+
-
- Aug 10, 2009
-
-
rtoy authored
o Standard streams no longer change formats when *default-external-format* changes. Use stream:set-system-external-format instead, or (setf external-format). o char-to-octets properly handles surrogate characters being written. o Makes simple-streams work again. This change needs to be cross-compiled. 2009-07 binaries work for cross-compiling using the 19e/boot-2008-05-cross-unicode-*.lisp cross-compile script.
-
- Jun 25, 2009
-
-
rtoy authored
o Handle the surrogate pairs in our UTF-16 strings.
-
- Jun 24, 2009
-
-
rtoy authored
-
- Jun 11, 2009
-
-
rtoy authored
unicode-utf16-extfmt-2009-06-11.
-