Skip to content
Snippets Groups Projects
  1. Nov 25, 2022
  2. Dec 16, 2021
  3. Aug 09, 2014
    • Raymond Toy's avatar
      Fix ticket #100 by implementing STREAM-FILE-POSITION · 9e687a21
      Raymond Toy authored
      Implements STREAM-FILE-POSiTION and (SETF STREAM-FILE-POSITION).
      
       * code/stream.lisp:
         * Add support for Gray streams in FILE-POSITION.
       * pcl/gray-streams.lisp:
         * Define STREAM-FILE-POSITION and (SETF STREAM-FILE-POSITION).
         * Add methods on FUNDAMENTAL-STREAM, CHARACTER-INPUT-STREAM, and
           CHARACTER-OUTPUT-STREAM.
       * code/exports.lisp:
         * Export STREAM-FILE-POSITION.
      9e687a21
  4. Aug 01, 2014
  5. Apr 21, 2012
    • Raymond Toy's avatar
      Fix bug in handling the state BOM marker and also extend to work · b040afc2
      Raymond Toy authored
      composing external formats.  The state BOM marker also needs to
      indicate how long the BOM is.
      
       * src/code/stream.lisp
         * Handle the BOM marker correctly for composed format. 
         * Handle the new values for the state BOM marker.
      
       * src/pcl/simple-streams/external-formats/utf-16.lisp
         * Change BOM state marker to +2 and -2 instead of 1 and 2 to
           indicate the length of the BOM.
      
       * src/pcl/simple-streams/external-formats/utf-32.lisp
         * Change BOM state marker to +4 and -4 instead of 1 and 2 to
           indicate the length of the BOM.
      b040afc2
  6. Apr 20, 2012
  7. Apr 19, 2012
    • Raymond Toy's avatar
      Fix ticket:58. Handle the BOM character for utf-16 and utf-32. This · f3db74d4
      Raymond Toy authored
      is a bit of a hack.
      
       * src/code/stream.lisp:
        * Check the state to see if a BOM was read.  This critically depends
          on knowing the format of the state variable for utf16 and utf32
          formats, but the stream code shouldn't have to know the state
          internals. 
      
       * src/general-info/release-20d.txt
         * Update.
      f3db74d4
  8. Nov 04, 2011
  9. Sep 25, 2011
  10. Jun 29, 2011
    • rtoy's avatar
      Fix Trac #43 again. · 5e7baddc
      rtoy authored
      stream.lisp:
      o Remove the old code that was the first attempt at fixing issue #43.
        This was wrong.
      
      fd-stream.lisp:
      o Fix issue #43.  Even if we have a string-buffer, we need to take
        into account any octets in the in-buffer (if it exists) that have
        not been processed.  This happens if the in-buffer does not have
        enough octets at the end to form a complete character for the given
        encoding.
      o Some debugging prints added, but disabled.
      5e7baddc
  11. Jun 27, 2011
    • rtoy's avatar
      Revert last change to fix Trac #43. · b948683f
      rtoy authored
      That changes causes the Unicode normalization and word-break tests to
      fail.  I do not know why this causes the tests to fail, but it causes
      the test program to suck in the entire file into a string instead of
      just one line.
      b948683f
  12. Mar 03, 2011
    • rtoy's avatar
      Fix Trac #43: unread-char doesn't change file-position · 89f8fa96
      rtoy authored
      The issue is caused by FAST-READ-CHAR-STRING-REFILL.  In some
      situations the number of octets converted is not exactly the same as
      the length of the buffer because the last octets in the buffer don't
      hold a complete encoded character.  This is ok, but we didn't update
      the ibuf-head pointer to indicate that some octets haven't actually
      been converted.  This confuses FILE-POSITION.
      89f8fa96
  13. Sep 08, 2010
    • rtoy's avatar
      Clean up how (setf stream-external-format) works. This change · 069f884d
      rtoy authored
      requires a cross-compile using the default cross-compile script.
      
      fd-stream.lisp:
      o Remove the SAVED-OC-STATE slot from an FD-STREAM because we don't
        need it anymore since we have the OCTET-COUNT slot.
      
      stream.lisp:
      o Don't need to copy the current OC-STATE to SAVED-OC-STATE.  Remove
        code and update comments.
      
      fd-stream-extfmt.lisp:
      o Use the OCTET-COUNT slot to figure out how many octets have been
        consumed so far to produce the characters that have already been
        read.  Don't need to do the re-conversion anymore, so we don't need
        the SAVED-OC-STATE anymore.
      o Add support for the case where we were using the ISO8859-1 external
        format and are now switching to another external format that
        requires the string-buffer.
      069f884d
  14. Sep 06, 2010
  15. Sep 04, 2010
  16. Aug 15, 2010
    • rtoy's avatar
      Fix file-position bug in trac #36. We add an array to keep track of · d689646f
      rtoy authored
      the octets consumed for each character.  This array is used to figure
      out the file position.  Some tests comparing this scheme indicates a
      very small slowdown of about 1%, so this seems not to hurt.
      
      Use a cross-compile using the 2010-07 snapshot to build this.  (Same
      procedure as used to build the 20b-pre1 release.)
      
      struct.lisp:
      o Add new slot OCTET-COUNT to LISP-STREAM to hold the array of octets
        per character.
      
      extfmts.lisp:
      o Add OCTETS-TO-STRING-COUNTED, which is like OCTETS-TO-STRING, except
        we need an array in which to store the number of octets consumed for
        each character processed.
      
      fd-stream.lisp:
      o Create the octet-count array creating the lisp stream string buffer.
      o In FD-STREAM-FILE-POSITION, use the octet count to count the number
        of octets that have been read but not yet returned to the user.
      
      stream.lisp:
      o Use OCTETS-TO-STRING-COUNTED instead of OCTETS-TO-STRING so we keep
        track of octet length of each character processed.
      d689646f
  17. Aug 09, 2010
  18. Jul 05, 2010
    • rtoy's avatar
      Fix critical error in fast-read-char-string-refill where we didn't · b92592e0
      rtoy authored
      stream the data correctly causing decoding errors.
      
      code/sysmacs.lisp:
      o Need to copy back the in-index that fast-read-char-string-refill
        updated.
      
      code/stream.lisp:
      o Fix buffering issue when refilling the in-buffer with new data.
        Code was confused about the difference between in-length and
        in-buffer-length.
      b92592e0
  19. Jul 02, 2010
    • rtoy's avatar
      Implement more of the external format error handlers. · f971e392
      rtoy authored
      code/extfmts.lisp
      o Call the error handler for iso8859-1 output.
      o In OCTETS-TO-CODEPOINT and CODEPOINT-TO-OCTETS, call the external
        format with the error argument.
      o In OCTETS-TO-CHAR
        - Call OCTETS-TO-CODEPOINT with the error handler.
        - For all of the error conditions, call the error handler if
          defined.
      o Add error parameter to EF-STRING-TO-OCTETS and EF-ENCODE so we can
        handle errors.  Call CHAR-TO-OCTETS with the error handler.
      o Add error parameter to STRING-TO-OCTETS and use it.
      o Add error parameter to EF-OCTETS-TO-STRING and EF-DECODE so we can
        handle errors.  Call OCTETS-TO-CHAR with the error handler.
      o Add error parameter to OCTETS-TO-STRING and use it.
      o In STRING-ENCODE and STRING-DECODE, call the ef function with the
        error handler.
      o Change STRING-ENCODE to use keyword args instead of optional args.
        Add error parameter and use it.
      
      code/fd-stream-extfmt.lisp:
      o Tell OCTETS-TO-STRING about the error handler stored in the
        fd-stream.
      
      code/fd-stream.lisp:
      o OPEN, MAKE-FD-STREAM, and OPEN-FD-STREAM get DECODING-ERROR and
        ENCODING-ERROR keyword arguments for specifying how to handle
        decoding and encoding errors in external formats.
      
      code/stream.lisp:
      o Make sure the error handler is called in
        FAST-READ-CHAR-STRING-REFILL.
      
      pcl/simple-streams/external-formats/utf-8.lisp:
      o Initial cut at calling the error handler for the various possible
        invalid octet streams for a utf-8 encoding.
      f971e392
  20. Apr 20, 2010
  21. Apr 19, 2010
  22. Mar 19, 2010
  23. Jan 23, 2010
    • rtoy's avatar
      Oops. Last change to fd-stream doesn't actually compile because no · 2ea523e6
      rtoy authored
      everything is defined yet.  Hence, add dummy
      %SET-FD-STREAM-EXTERNAL-FORMAT and move the real one to
      fd-stream-extfmt.lisp.  This builds.
      
      code/fd-stream.lisp:
      o Always call %SET-FD-STREAM-EXTERNAL-FORMAT, even if
        LISP::*ENABLE-STREAM-BUFFER-P* is NIL.
      
      code/stream.lisp:
      o Move %SET-FD-STREAM-EXTERNAL-FORMAT to fd-stream-extfmt.lisp.
      o Add dummy implementation of %SET-FD-STREAM-EXTERNAL-FORMAT.
      
      code/fd-stream-extfmt.lisp:
      o %SET-FD-STREAM-EXTERNAL-FORMAT moved here.
      2ea523e6
  24. Oct 18, 2009
    • rtoy's avatar
      Merge changes from unicode-string-buffer-impl-branch which gives · 392d3e59
      rtoy authored
      faster reads on external-formats.  This is done by adding an
      additional buffer to streams so we can convert the entire in-buffer
      into characters all at once.
      
      To build this change, you need to do a cross-compile using
      boot-2009-10-1-cross.lisp.  Using that build, do a normal build with
      these sources.
      
      For a non-unicode build use boot-2009-10-01.lisp with a 20a
      non-unicode build.
      
      code/extfmts.lisp:
      o Add another slot to the extfmts for copying the state.
      o Modify EF-OCTETS-TO-STRING and OCTETS-TO-STRING to support the
        necesssary changes for fast formats.  This is incompatible with the
        previous version because the string is not grown if needed.
      
      code/fd-stream-extfmt.lisp:
      o Set *enable-stream-buffer-p* to T so we have fast streams.
      
      code/fd-stream.lisp:
      o Add new slots to support fast strams.
      o In SET-ROUTINES, initialize the new slots appropriately.
      o Update UNREAD-CHAR to be able to back up in the string buffer to
        unread.
      o Add implementation to copy the state of an external format.
      
      code/stream.lisp:
      o Change %SET-FD-STREAM-EXTERNAL-FORMAT to be able to change formats
        even if we've already converted the buffer with a different format.
        We reconvert the buffer with the old format until we reach the
        current character.  Then the remaining octets are converted using
        the new format and stored in the string buffer.
      o Add FAST-READ-CHAR-STRING-REFILL to refill the string buffer, like
        FAST-READ-CHAR-REFILL does for the octet in-buffer.
      
      code/struct.lisp:
      o Add new slots to hold the string buffer, the current index, and
        length.  These are needed for the fast formats.
      
      code/sysmacs.lisp:
      o Update PREPARE-FOR-FAST-READ-CHAR, DONE-WITH-FAST-READ-CHAR, and
        FAST-READ-CHAR to support the string buffer.
      
      code/string.lisp:
      o Microoptimization of SURROGATEP to reduce the number of branchs.
      
      general-info/release-20b.txt:
      o Update with these changes
      
      pcl/simple-streams/external-formats/utf-16-be.lisp:
      pcl/simple-streams/external-formats/utf-16-le.lisp:
      pcl/simple-streams/external-formats/utf-16.lisp:
      o These formats actually have state, so update them to take a handle
        an initial state.  These are needed if the string buffer ends with a
        leading surrogate and the next string buffer starts with a trailing
        surrogate.  The conversion needs to combine the surrogates together.
      392d3e59
  25. Aug 10, 2009
    • rtoy's avatar
      Fixes from Paul Foley: · 94ed0111
      rtoy authored
      o Standard streams no longer change formats when
        *default-external-format* changes.  Use
        stream:set-system-external-format instead, or (setf
        external-format).
      o char-to-octets properly handles surrogate characters being written.
      o Makes simple-streams work again.
      
      This change needs to be cross-compiled.  2009-07 binaries work for
      cross-compiling using the 19e/boot-2008-05-cross-unicode-*.lisp
      cross-compile script.
      94ed0111
  26. Jul 17, 2009
  27. Jun 16, 2009
    • rtoy's avatar
      Cleanups for non-unicode build. · 8f28c28f
      rtoy authored
      code/stream.lisp:
      o Only define (setf stream-external-format) for Unicode builds.
      o In stream-external-format, don't try to look up the external format
        from the fd-stream structure, which doesn't exist in non-unicode
        builds.
      
      code/strings.lisp:
      o Conditionalize out things that will only work if unicode is
        available.
      
      tools/worldcom.lisp:
      o Only compile fd-stream-extfmt for unicode builds.
      8f28c28f
  28. Jun 11, 2009
  29. Aug 21, 2006
    • rtoy's avatar
      Add BASE-CHAR and CHARACTER to · 8bfbe53c
      rtoy authored
      *READ-INTO-SIMPLE-ARRAY-RECOGNIZED-TYPES* so that READ-VECTOR can read
      binary data from streams of those types.  Fixes Trace ticket 7.
      8bfbe53c
  30. Nov 11, 2005
  31. Jul 01, 2005
  32. May 23, 2005
    • rtoy's avatar
      Bug from cmucl-imp, 2005-05-17 by Fredrik Sandstrom: · 668b13f5
      rtoy authored
          Description: In (peek-char nil s nil foo), if foo happens to be the
          same character that peek-char returns, the character is removed from
          the input stream, as if read by read-char.
      
          Examples:
      
          * (with-input-from-string (s "123")
      	(list (peek-char nil s nil #\1) (read-char s) (read-char s)))
          (#\1 #\2 #\3)
      
      This fix is based on the version proposed by Rudi Schlatte, with minor
      changes in naming.
      668b13f5
  33. Apr 19, 2005
    • rtoy's avatar
      Rob Warnock reports on c.l.l that · bdb3e53c
      rtoy authored
      (let ((s (make-two-way-stream (make-concatenated-stream)
      				  (make-broadcast-stream))))
            (dotimes (i 10) (format s "Hello, i = ~a~%" i)))
      
      doesn't work.  This is caused by TWO-WAY-MISC not handling the
      :CHARPOS and :LINE-LENGTH operations.  Just call the appropriate
      versions for the output stream of the two-way stream.
      bdb3e53c
  34. Feb 21, 2005
    • rtoy's avatar
      Apply Lynn Quam's proposed enhancements to fd-streams allowing input · d4ad6755
      rtoy authored
      streams opened with element-type (unsigned-byte 8) to be also opened
      for character input.  This gives simple-streams semantics to
      fd-streams.
      
      * code/fd-stream.lisp
        o Make FD-STREAM-IN use PICK-INPUT-ROUTINE to select the correct
          type of input.
        o Add *FD-STREAM-ENABLE-CHARACTER-AND-BINARY-INPUT* to allow binary
          and character input.  Currently defaults to NIL, but will change
          to T.
      
      * code/stream.lisp
        o New READ-INTO-SIMPLE-STRING to allow stream element-type of
          (unsigned-byte 8).
        o Support reading and writing simple arrays of single-float and
          double-float.
        o Strings can be written to streams of element-type '(unsigned-byte
          8).
      
      * tools/worldcom.lisp
        o Compile stream-vector-io.lisp.
      
      * tools/worldload.lisp
        o Load stream-vector-io.
      
      * code/stream-vector-io.lisp
        o New file implementing READ-VECTOR and WRITE-VECTOR.
      d4ad6755
  35. Apr 16, 2004
  36. Apr 15, 2004
    • rtoy's avatar
      Fix ansi-test bug on read-sequence and echo-streams. Based on code · 48087ddd
      rtoy authored
      from SBCL.
      48087ddd
    • rtoy's avatar
      More ANSI test fixes: · 2e51cd52
      rtoy authored
      o FILE-STRING-LENGTH can be used on BROADCAST-STREAM's where the
        result is 1.
      o Graphic characters are not printed using the character name (hence
        #\space is printed #\ )
      o Make INPUT-STREAM-P and OUTPUT-STREAM-P work correctly on synonym
        streams.
      o MAKE-STRING-INPUT-STREAM was not accepting NIL for the optional END
        argument.
      o Close string-input streams.  (We weren't before.)
      2e51cd52
  37. Apr 14, 2004
  38. Apr 09, 2004
    • emarsden's avatar
      · 3f2ba723
      emarsden authored
      Fix stupid mistake in the broadcast-stream ANSI-compliance changes.
      3f2ba723
Loading