- Nov 25, 2022
-
-
Raymond Toy authored
-
- Dec 16, 2021
-
-
Raymond Toy authored
-
- Aug 09, 2014
-
-
Raymond Toy authored
Implements STREAM-FILE-POSiTION and (SETF STREAM-FILE-POSITION). * code/stream.lisp: * Add support for Gray streams in FILE-POSITION. * pcl/gray-streams.lisp: * Define STREAM-FILE-POSITION and (SETF STREAM-FILE-POSITION). * Add methods on FUNDAMENTAL-STREAM, CHARACTER-INPUT-STREAM, and CHARACTER-OUTPUT-STREAM. * code/exports.lisp: * Export STREAM-FILE-POSITION.
-
- Aug 01, 2014
-
-
Raymond Toy authored
See ticket #101. This fixes item 1 of that bug report.
-
- Apr 21, 2012
-
-
Raymond Toy authored
composing external formats. The state BOM marker also needs to indicate how long the BOM is. * src/code/stream.lisp * Handle the BOM marker correctly for composed format. * Handle the new values for the state BOM marker. * src/pcl/simple-streams/external-formats/utf-16.lisp * Change BOM state marker to +2 and -2 instead of 1 and 2 to indicate the length of the BOM. * src/pcl/simple-streams/external-formats/utf-32.lisp * Change BOM state marker to +4 and -4 instead of 1 and 2 to indicate the length of the BOM.
-
- Apr 20, 2012
-
-
Raymond Toy authored
-
Raymond Toy authored
-
- Apr 19, 2012
-
-
Raymond Toy authored
is a bit of a hack. * src/code/stream.lisp: * Check the state to see if a BOM was read. This critically depends on knowing the format of the state variable for utf16 and utf32 formats, but the stream code shouldn't have to know the state internals. * src/general-info/release-20d.txt * Update.
-
- Nov 04, 2011
-
-
Raymond Toy authored
-
- Sep 25, 2011
-
-
Raymond Toy authored
entries with just the file path, removing the revision number, date, author and state. The actual information is now computed during compilation and stored in the fasl itself. (See ticket:48.)
-
- Jun 29, 2011
-
-
rtoy authored
stream.lisp: o Remove the old code that was the first attempt at fixing issue #43. This was wrong. fd-stream.lisp: o Fix issue #43. Even if we have a string-buffer, we need to take into account any octets in the in-buffer (if it exists) that have not been processed. This happens if the in-buffer does not have enough octets at the end to form a complete character for the given encoding. o Some debugging prints added, but disabled.
-
- Jun 27, 2011
-
-
rtoy authored
That changes causes the Unicode normalization and word-break tests to fail. I do not know why this causes the tests to fail, but it causes the test program to suck in the entire file into a string instead of just one line.
-
- Mar 03, 2011
-
-
rtoy authored
The issue is caused by FAST-READ-CHAR-STRING-REFILL. In some situations the number of octets converted is not exactly the same as the length of the buffer because the last octets in the buffer don't hold a complete encoded character. This is ok, but we didn't update the ibuf-head pointer to indicate that some octets haven't actually been converted. This confuses FILE-POSITION.
-
- Sep 08, 2010
-
-
rtoy authored
requires a cross-compile using the default cross-compile script. fd-stream.lisp: o Remove the SAVED-OC-STATE slot from an FD-STREAM because we don't need it anymore since we have the OCTET-COUNT slot. stream.lisp: o Don't need to copy the current OC-STATE to SAVED-OC-STATE. Remove code and update comments. fd-stream-extfmt.lisp: o Use the OCTET-COUNT slot to figure out how many octets have been consumed so far to produce the characters that have already been read. Don't need to do the re-conversion anymore, so we don't need the SAVED-OC-STATE anymore. o Add support for the case where we were using the ISO8859-1 external format and are now switching to another external format that requires the string-buffer.
-
- Sep 06, 2010
-
-
rtoy authored
-
- Sep 04, 2010
-
-
rtoy authored
-
- Aug 15, 2010
-
-
rtoy authored
the octets consumed for each character. This array is used to figure out the file position. Some tests comparing this scheme indicates a very small slowdown of about 1%, so this seems not to hurt. Use a cross-compile using the 2010-07 snapshot to build this. (Same procedure as used to build the 20b-pre1 release.) struct.lisp: o Add new slot OCTET-COUNT to LISP-STREAM to hold the array of octets per character. extfmts.lisp: o Add OCTETS-TO-STRING-COUNTED, which is like OCTETS-TO-STRING, except we need an array in which to store the number of octets consumed for each character processed. fd-stream.lisp: o Create the octet-count array creating the lisp stream string buffer. o In FD-STREAM-FILE-POSITION, use the octet count to count the number of octets that have been read but not yet returned to the user. stream.lisp: o Use OCTETS-TO-STRING-COUNTED instead of OCTETS-TO-STRING so we keep track of octet length of each character processed.
-
- Aug 09, 2010
-
-
rtoy authored
LISP-STREAM-MISC function takes care of computing the position, so FILE-POSITION doesn't need to do anything else. (Why isn't this true for the non-unicode case? Should we make it so?)
-
- Jul 05, 2010
-
-
rtoy authored
stream the data correctly causing decoding errors. code/sysmacs.lisp: o Need to copy back the in-index that fast-read-char-string-refill updated. code/stream.lisp: o Fix buffering issue when refilling the in-buffer with new data. Code was confused about the difference between in-length and in-buffer-length.
-
- Jul 02, 2010
-
-
rtoy authored
code/extfmts.lisp o Call the error handler for iso8859-1 output. o In OCTETS-TO-CODEPOINT and CODEPOINT-TO-OCTETS, call the external format with the error argument. o In OCTETS-TO-CHAR - Call OCTETS-TO-CODEPOINT with the error handler. - For all of the error conditions, call the error handler if defined. o Add error parameter to EF-STRING-TO-OCTETS and EF-ENCODE so we can handle errors. Call CHAR-TO-OCTETS with the error handler. o Add error parameter to STRING-TO-OCTETS and use it. o Add error parameter to EF-OCTETS-TO-STRING and EF-DECODE so we can handle errors. Call OCTETS-TO-CHAR with the error handler. o Add error parameter to OCTETS-TO-STRING and use it. o In STRING-ENCODE and STRING-DECODE, call the ef function with the error handler. o Change STRING-ENCODE to use keyword args instead of optional args. Add error parameter and use it. code/fd-stream-extfmt.lisp: o Tell OCTETS-TO-STRING about the error handler stored in the fd-stream. code/fd-stream.lisp: o OPEN, MAKE-FD-STREAM, and OPEN-FD-STREAM get DECODING-ERROR and ENCODING-ERROR keyword arguments for specifying how to handle decoding and encoding errors in external formats. code/stream.lisp: o Make sure the error handler is called in FAST-READ-CHAR-STRING-REFILL. pcl/simple-streams/external-formats/utf-8.lisp: o Initial cut at calling the error handler for the various possible invalid octet streams for a utf-8 encoding.
-
- Apr 20, 2010
-
-
rtoy authored
may get confused with source locations if the reader macros are installed.
-
- Apr 19, 2010
-
-
rtoy authored
-
- Mar 19, 2010
-
-
rtoy authored
boot-2010-02-1 as the bootstrap file. You should probably also use the new -P option for build.sh to generate and update the po files while building.
-
- Jan 23, 2010
-
-
rtoy authored
everything is defined yet. Hence, add dummy %SET-FD-STREAM-EXTERNAL-FORMAT and move the real one to fd-stream-extfmt.lisp. This builds. code/fd-stream.lisp: o Always call %SET-FD-STREAM-EXTERNAL-FORMAT, even if LISP::*ENABLE-STREAM-BUFFER-P* is NIL. code/stream.lisp: o Move %SET-FD-STREAM-EXTERNAL-FORMAT to fd-stream-extfmt.lisp. o Add dummy implementation of %SET-FD-STREAM-EXTERNAL-FORMAT. code/fd-stream-extfmt.lisp: o %SET-FD-STREAM-EXTERNAL-FORMAT moved here.
-
- Oct 18, 2009
-
-
rtoy authored
faster reads on external-formats. This is done by adding an additional buffer to streams so we can convert the entire in-buffer into characters all at once. To build this change, you need to do a cross-compile using boot-2009-10-1-cross.lisp. Using that build, do a normal build with these sources. For a non-unicode build use boot-2009-10-01.lisp with a 20a non-unicode build. code/extfmts.lisp: o Add another slot to the extfmts for copying the state. o Modify EF-OCTETS-TO-STRING and OCTETS-TO-STRING to support the necesssary changes for fast formats. This is incompatible with the previous version because the string is not grown if needed. code/fd-stream-extfmt.lisp: o Set *enable-stream-buffer-p* to T so we have fast streams. code/fd-stream.lisp: o Add new slots to support fast strams. o In SET-ROUTINES, initialize the new slots appropriately. o Update UNREAD-CHAR to be able to back up in the string buffer to unread. o Add implementation to copy the state of an external format. code/stream.lisp: o Change %SET-FD-STREAM-EXTERNAL-FORMAT to be able to change formats even if we've already converted the buffer with a different format. We reconvert the buffer with the old format until we reach the current character. Then the remaining octets are converted using the new format and stored in the string buffer. o Add FAST-READ-CHAR-STRING-REFILL to refill the string buffer, like FAST-READ-CHAR-REFILL does for the octet in-buffer. code/struct.lisp: o Add new slots to hold the string buffer, the current index, and length. These are needed for the fast formats. code/sysmacs.lisp: o Update PREPARE-FOR-FAST-READ-CHAR, DONE-WITH-FAST-READ-CHAR, and FAST-READ-CHAR to support the string buffer. code/string.lisp: o Microoptimization of SURROGATEP to reduce the number of branchs. general-info/release-20b.txt: o Update with these changes pcl/simple-streams/external-formats/utf-16-be.lisp: pcl/simple-streams/external-formats/utf-16-le.lisp: pcl/simple-streams/external-formats/utf-16.lisp: o These formats actually have state, so update them to take a handle an initial state. These are needed if the string buffer ends with a leading surrogate and the next string buffer starts with a trailing surrogate. The conversion needs to combine the surrogates together.
-
- Aug 10, 2009
-
-
rtoy authored
o Standard streams no longer change formats when *default-external-format* changes. Use stream:set-system-external-format instead, or (setf external-format). o char-to-octets properly handles surrogate characters being written. o Makes simple-streams work again. This change needs to be cross-compiled. 2009-07 binaries work for cross-compiling using the 19e/boot-2008-05-cross-unicode-*.lisp cross-compile script.
-
- Jul 17, 2009
-
-
rtoy authored
docstrings, especially when displayed with describe.
-
- Jun 16, 2009
-
-
rtoy authored
code/stream.lisp: o Only define (setf stream-external-format) for Unicode builds. o In stream-external-format, don't try to look up the external format from the fd-stream structure, which doesn't exist in non-unicode builds. code/strings.lisp: o Conditionalize out things that will only work if unicode is available. tools/worldcom.lisp: o Only compile fd-stream-extfmt for unicode builds.
-
- Jun 11, 2009
-
-
rtoy authored
unicode-utf16-extfmt-2009-06-11.
-
- Aug 21, 2006
-
-
rtoy authored
*READ-INTO-SIMPLE-ARRAY-RECOGNIZED-TYPES* so that READ-VECTOR can read binary data from streams of those types. Fixes Trace ticket 7.
-
- Nov 11, 2005
-
-
rtoy authored
cmucl-imp, 2005-11-10, with fix. o Apply Walter's fix, because it was missing the fifth arg to GENERALIZED-PEEKING-MECHANISM. o Change GENERALIZED-PEEKING-MACHINISM to use keyword args instead of optional args, to prevent this kind of bug.
-
- Jul 01, 2005
-
-
rtoy authored
for the position. Add support. Reported by Bruno Haible, cmucl-imp, 28 Jun 2005.
-
- May 23, 2005
-
-
rtoy authored
Description: In (peek-char nil s nil foo), if foo happens to be the same character that peek-char returns, the character is removed from the input stream, as if read by read-char. Examples: * (with-input-from-string (s "123") (list (peek-char nil s nil #\1) (read-char s) (read-char s))) (#\1 #\2 #\3) This fix is based on the version proposed by Rudi Schlatte, with minor changes in naming.
-
- Apr 19, 2005
-
-
rtoy authored
(let ((s (make-two-way-stream (make-concatenated-stream) (make-broadcast-stream)))) (dotimes (i 10) (format s "Hello, i = ~a~%" i))) doesn't work. This is caused by TWO-WAY-MISC not handling the :CHARPOS and :LINE-LENGTH operations. Just call the appropriate versions for the output stream of the two-way stream.
-
- Feb 21, 2005
-
-
rtoy authored
streams opened with element-type (unsigned-byte 8) to be also opened for character input. This gives simple-streams semantics to fd-streams. * code/fd-stream.lisp o Make FD-STREAM-IN use PICK-INPUT-ROUTINE to select the correct type of input. o Add *FD-STREAM-ENABLE-CHARACTER-AND-BINARY-INPUT* to allow binary and character input. Currently defaults to NIL, but will change to T. * code/stream.lisp o New READ-INTO-SIMPLE-STRING to allow stream element-type of (unsigned-byte 8). o Support reading and writing simple arrays of single-float and double-float. o Strings can be written to streams of element-type '(unsigned-byte 8). * tools/worldcom.lisp o Compile stream-vector-io.lisp. * tools/worldload.lisp o Load stream-vector-io. * code/stream-vector-io.lisp o New file implementing READ-VECTOR and WRITE-VECTOR.
-
- Apr 16, 2004
-
-
rtoy authored
-
- Apr 15, 2004
-
-
rtoy authored
from SBCL.
-
rtoy authored
o FILE-STRING-LENGTH can be used on BROADCAST-STREAM's where the result is 1. o Graphic characters are not printed using the character name (hence #\space is printed #\ ) o Make INPUT-STREAM-P and OUTPUT-STREAM-P work correctly on synonym streams. o MAKE-STRING-INPUT-STREAM was not accepting NIL for the optional END argument. o Close string-input streams. (We weren't before.)
-
- Apr 14, 2004
-
-
rtoy authored
TWO-WAY, CONCATENATED, ECHO, STRING-INPUT, and STRING-OUTPUT CLHS says these streams produce errors for file-length.
-
- Apr 09, 2004
-
-
emarsden authored
Fix stupid mistake in the broadcast-stream ANSI-compliance changes.
-