Skip to content
Snippets Groups Projects
  1. Nov 04, 2011
  2. Sep 25, 2011
  3. Jun 29, 2011
    • rtoy's avatar
      Fix Trac #43 again. · 5e7baddc
      rtoy authored
      stream.lisp:
      o Remove the old code that was the first attempt at fixing issue #43.
        This was wrong.
      
      fd-stream.lisp:
      o Fix issue #43.  Even if we have a string-buffer, we need to take
        into account any octets in the in-buffer (if it exists) that have
        not been processed.  This happens if the in-buffer does not have
        enough octets at the end to form a complete character for the given
        encoding.
      o Some debugging prints added, but disabled.
      5e7baddc
  4. Jun 27, 2011
    • rtoy's avatar
      Revert last change to fix Trac #43. · b948683f
      rtoy authored
      That changes causes the Unicode normalization and word-break tests to
      fail.  I do not know why this causes the tests to fail, but it causes
      the test program to suck in the entire file into a string instead of
      just one line.
      b948683f
  5. Mar 03, 2011
    • rtoy's avatar
      Fix Trac #43: unread-char doesn't change file-position · 89f8fa96
      rtoy authored
      The issue is caused by FAST-READ-CHAR-STRING-REFILL.  In some
      situations the number of octets converted is not exactly the same as
      the length of the buffer because the last octets in the buffer don't
      hold a complete encoded character.  This is ok, but we didn't update
      the ibuf-head pointer to indicate that some octets haven't actually
      been converted.  This confuses FILE-POSITION.
      89f8fa96
  6. Sep 08, 2010
    • rtoy's avatar
      Clean up how (setf stream-external-format) works. This change · 069f884d
      rtoy authored
      requires a cross-compile using the default cross-compile script.
      
      fd-stream.lisp:
      o Remove the SAVED-OC-STATE slot from an FD-STREAM because we don't
        need it anymore since we have the OCTET-COUNT slot.
      
      stream.lisp:
      o Don't need to copy the current OC-STATE to SAVED-OC-STATE.  Remove
        code and update comments.
      
      fd-stream-extfmt.lisp:
      o Use the OCTET-COUNT slot to figure out how many octets have been
        consumed so far to produce the characters that have already been
        read.  Don't need to do the re-conversion anymore, so we don't need
        the SAVED-OC-STATE anymore.
      o Add support for the case where we were using the ISO8859-1 external
        format and are now switching to another external format that
        requires the string-buffer.
      069f884d
  7. Sep 06, 2010
  8. Sep 04, 2010
  9. Aug 15, 2010
    • rtoy's avatar
      Fix file-position bug in trac #36. We add an array to keep track of · d689646f
      rtoy authored
      the octets consumed for each character.  This array is used to figure
      out the file position.  Some tests comparing this scheme indicates a
      very small slowdown of about 1%, so this seems not to hurt.
      
      Use a cross-compile using the 2010-07 snapshot to build this.  (Same
      procedure as used to build the 20b-pre1 release.)
      
      struct.lisp:
      o Add new slot OCTET-COUNT to LISP-STREAM to hold the array of octets
        per character.
      
      extfmts.lisp:
      o Add OCTETS-TO-STRING-COUNTED, which is like OCTETS-TO-STRING, except
        we need an array in which to store the number of octets consumed for
        each character processed.
      
      fd-stream.lisp:
      o Create the octet-count array creating the lisp stream string buffer.
      o In FD-STREAM-FILE-POSITION, use the octet count to count the number
        of octets that have been read but not yet returned to the user.
      
      stream.lisp:
      o Use OCTETS-TO-STRING-COUNTED instead of OCTETS-TO-STRING so we keep
        track of octet length of each character processed.
      d689646f
  10. Aug 09, 2010
  11. Jul 05, 2010
    • rtoy's avatar
      Fix critical error in fast-read-char-string-refill where we didn't · b92592e0
      rtoy authored
      stream the data correctly causing decoding errors.
      
      code/sysmacs.lisp:
      o Need to copy back the in-index that fast-read-char-string-refill
        updated.
      
      code/stream.lisp:
      o Fix buffering issue when refilling the in-buffer with new data.
        Code was confused about the difference between in-length and
        in-buffer-length.
      b92592e0
  12. Jul 02, 2010
    • rtoy's avatar
      Implement more of the external format error handlers. · f971e392
      rtoy authored
      code/extfmts.lisp
      o Call the error handler for iso8859-1 output.
      o In OCTETS-TO-CODEPOINT and CODEPOINT-TO-OCTETS, call the external
        format with the error argument.
      o In OCTETS-TO-CHAR
        - Call OCTETS-TO-CODEPOINT with the error handler.
        - For all of the error conditions, call the error handler if
          defined.
      o Add error parameter to EF-STRING-TO-OCTETS and EF-ENCODE so we can
        handle errors.  Call CHAR-TO-OCTETS with the error handler.
      o Add error parameter to STRING-TO-OCTETS and use it.
      o Add error parameter to EF-OCTETS-TO-STRING and EF-DECODE so we can
        handle errors.  Call OCTETS-TO-CHAR with the error handler.
      o Add error parameter to OCTETS-TO-STRING and use it.
      o In STRING-ENCODE and STRING-DECODE, call the ef function with the
        error handler.
      o Change STRING-ENCODE to use keyword args instead of optional args.
        Add error parameter and use it.
      
      code/fd-stream-extfmt.lisp:
      o Tell OCTETS-TO-STRING about the error handler stored in the
        fd-stream.
      
      code/fd-stream.lisp:
      o OPEN, MAKE-FD-STREAM, and OPEN-FD-STREAM get DECODING-ERROR and
        ENCODING-ERROR keyword arguments for specifying how to handle
        decoding and encoding errors in external formats.
      
      code/stream.lisp:
      o Make sure the error handler is called in
        FAST-READ-CHAR-STRING-REFILL.
      
      pcl/simple-streams/external-formats/utf-8.lisp:
      o Initial cut at calling the error handler for the various possible
        invalid octet streams for a utf-8 encoding.
      f971e392
  13. Apr 20, 2010
  14. Apr 19, 2010
  15. Mar 19, 2010
  16. Jan 23, 2010
    • rtoy's avatar
      Oops. Last change to fd-stream doesn't actually compile because no · 2ea523e6
      rtoy authored
      everything is defined yet.  Hence, add dummy
      %SET-FD-STREAM-EXTERNAL-FORMAT and move the real one to
      fd-stream-extfmt.lisp.  This builds.
      
      code/fd-stream.lisp:
      o Always call %SET-FD-STREAM-EXTERNAL-FORMAT, even if
        LISP::*ENABLE-STREAM-BUFFER-P* is NIL.
      
      code/stream.lisp:
      o Move %SET-FD-STREAM-EXTERNAL-FORMAT to fd-stream-extfmt.lisp.
      o Add dummy implementation of %SET-FD-STREAM-EXTERNAL-FORMAT.
      
      code/fd-stream-extfmt.lisp:
      o %SET-FD-STREAM-EXTERNAL-FORMAT moved here.
      2ea523e6
  17. Oct 18, 2009
    • rtoy's avatar
      Merge changes from unicode-string-buffer-impl-branch which gives · 392d3e59
      rtoy authored
      faster reads on external-formats.  This is done by adding an
      additional buffer to streams so we can convert the entire in-buffer
      into characters all at once.
      
      To build this change, you need to do a cross-compile using
      boot-2009-10-1-cross.lisp.  Using that build, do a normal build with
      these sources.
      
      For a non-unicode build use boot-2009-10-01.lisp with a 20a
      non-unicode build.
      
      code/extfmts.lisp:
      o Add another slot to the extfmts for copying the state.
      o Modify EF-OCTETS-TO-STRING and OCTETS-TO-STRING to support the
        necesssary changes for fast formats.  This is incompatible with the
        previous version because the string is not grown if needed.
      
      code/fd-stream-extfmt.lisp:
      o Set *enable-stream-buffer-p* to T so we have fast streams.
      
      code/fd-stream.lisp:
      o Add new slots to support fast strams.
      o In SET-ROUTINES, initialize the new slots appropriately.
      o Update UNREAD-CHAR to be able to back up in the string buffer to
        unread.
      o Add implementation to copy the state of an external format.
      
      code/stream.lisp:
      o Change %SET-FD-STREAM-EXTERNAL-FORMAT to be able to change formats
        even if we've already converted the buffer with a different format.
        We reconvert the buffer with the old format until we reach the
        current character.  Then the remaining octets are converted using
        the new format and stored in the string buffer.
      o Add FAST-READ-CHAR-STRING-REFILL to refill the string buffer, like
        FAST-READ-CHAR-REFILL does for the octet in-buffer.
      
      code/struct.lisp:
      o Add new slots to hold the string buffer, the current index, and
        length.  These are needed for the fast formats.
      
      code/sysmacs.lisp:
      o Update PREPARE-FOR-FAST-READ-CHAR, DONE-WITH-FAST-READ-CHAR, and
        FAST-READ-CHAR to support the string buffer.
      
      code/string.lisp:
      o Microoptimization of SURROGATEP to reduce the number of branchs.
      
      general-info/release-20b.txt:
      o Update with these changes
      
      pcl/simple-streams/external-formats/utf-16-be.lisp:
      pcl/simple-streams/external-formats/utf-16-le.lisp:
      pcl/simple-streams/external-formats/utf-16.lisp:
      o These formats actually have state, so update them to take a handle
        an initial state.  These are needed if the string buffer ends with a
        leading surrogate and the next string buffer starts with a trailing
        surrogate.  The conversion needs to combine the surrogates together.
      392d3e59
  18. Aug 10, 2009
    • rtoy's avatar
      Fixes from Paul Foley: · 94ed0111
      rtoy authored
      o Standard streams no longer change formats when
        *default-external-format* changes.  Use
        stream:set-system-external-format instead, or (setf
        external-format).
      o char-to-octets properly handles surrogate characters being written.
      o Makes simple-streams work again.
      
      This change needs to be cross-compiled.  2009-07 binaries work for
      cross-compiling using the 19e/boot-2008-05-cross-unicode-*.lisp
      cross-compile script.
      94ed0111
  19. Jul 17, 2009
  20. Jun 16, 2009
    • rtoy's avatar
      Cleanups for non-unicode build. · 8f28c28f
      rtoy authored
      code/stream.lisp:
      o Only define (setf stream-external-format) for Unicode builds.
      o In stream-external-format, don't try to look up the external format
        from the fd-stream structure, which doesn't exist in non-unicode
        builds.
      
      code/strings.lisp:
      o Conditionalize out things that will only work if unicode is
        available.
      
      tools/worldcom.lisp:
      o Only compile fd-stream-extfmt for unicode builds.
      8f28c28f
  21. Jun 11, 2009
  22. Aug 21, 2006
    • rtoy's avatar
      Add BASE-CHAR and CHARACTER to · 8bfbe53c
      rtoy authored
      *READ-INTO-SIMPLE-ARRAY-RECOGNIZED-TYPES* so that READ-VECTOR can read
      binary data from streams of those types.  Fixes Trace ticket 7.
      8bfbe53c
  23. Nov 11, 2005
  24. Jul 01, 2005
  25. May 23, 2005
    • rtoy's avatar
      Bug from cmucl-imp, 2005-05-17 by Fredrik Sandstrom: · 668b13f5
      rtoy authored
          Description: In (peek-char nil s nil foo), if foo happens to be the
          same character that peek-char returns, the character is removed from
          the input stream, as if read by read-char.
      
          Examples:
      
          * (with-input-from-string (s "123")
      	(list (peek-char nil s nil #\1) (read-char s) (read-char s)))
          (#\1 #\2 #\3)
      
      This fix is based on the version proposed by Rudi Schlatte, with minor
      changes in naming.
      668b13f5
  26. Apr 19, 2005
    • rtoy's avatar
      Rob Warnock reports on c.l.l that · bdb3e53c
      rtoy authored
      (let ((s (make-two-way-stream (make-concatenated-stream)
      				  (make-broadcast-stream))))
            (dotimes (i 10) (format s "Hello, i = ~a~%" i)))
      
      doesn't work.  This is caused by TWO-WAY-MISC not handling the
      :CHARPOS and :LINE-LENGTH operations.  Just call the appropriate
      versions for the output stream of the two-way stream.
      bdb3e53c
  27. Feb 21, 2005
    • rtoy's avatar
      Apply Lynn Quam's proposed enhancements to fd-streams allowing input · d4ad6755
      rtoy authored
      streams opened with element-type (unsigned-byte 8) to be also opened
      for character input.  This gives simple-streams semantics to
      fd-streams.
      
      * code/fd-stream.lisp
        o Make FD-STREAM-IN use PICK-INPUT-ROUTINE to select the correct
          type of input.
        o Add *FD-STREAM-ENABLE-CHARACTER-AND-BINARY-INPUT* to allow binary
          and character input.  Currently defaults to NIL, but will change
          to T.
      
      * code/stream.lisp
        o New READ-INTO-SIMPLE-STRING to allow stream element-type of
          (unsigned-byte 8).
        o Support reading and writing simple arrays of single-float and
          double-float.
        o Strings can be written to streams of element-type '(unsigned-byte
          8).
      
      * tools/worldcom.lisp
        o Compile stream-vector-io.lisp.
      
      * tools/worldload.lisp
        o Load stream-vector-io.
      
      * code/stream-vector-io.lisp
        o New file implementing READ-VECTOR and WRITE-VECTOR.
      d4ad6755
  28. Apr 16, 2004
  29. Apr 15, 2004
    • rtoy's avatar
      Fix ansi-test bug on read-sequence and echo-streams. Based on code · 48087ddd
      rtoy authored
      from SBCL.
      48087ddd
    • rtoy's avatar
      More ANSI test fixes: · 2e51cd52
      rtoy authored
      o FILE-STRING-LENGTH can be used on BROADCAST-STREAM's where the
        result is 1.
      o Graphic characters are not printed using the character name (hence
        #\space is printed #\ )
      o Make INPUT-STREAM-P and OUTPUT-STREAM-P work correctly on synonym
        streams.
      o MAKE-STRING-INPUT-STREAM was not accepting NIL for the optional END
        argument.
      o Close string-input streams.  (We weren't before.)
      2e51cd52
  30. Apr 14, 2004
  31. Apr 09, 2004
    • emarsden's avatar
      · 3f2ba723
      emarsden authored
      Fix stupid mistake in the broadcast-stream ANSI-compliance changes.
      3f2ba723
  32. Apr 07, 2004
    • emarsden's avatar
      ANSI compliance fixes for CONCATENATED-STREAMS: · c7aa07e6
      emarsden authored
        - (read-char-no-hang (make-concatenated-stream) nil :eof) returns :eof
          instead of nil
      
      Another bug brought to you by pfdietz.
      c7aa07e6
    • emarsden's avatar
      · 3eeefd6b
      emarsden authored
      More ANSI-compliance BROADCAST-STREAM fixes:
      
        - FILE-LENGTH and FILE-POSITION return the value from the last component
          stream, or 0 if there are no component streams.
      
      Will require a similar change for FILE-STRING-LENGTH (and addition of a
      :file-string-length misc-op).
      3eeefd6b
  33. Apr 06, 2004
    • emarsden's avatar
      More ANSI compliance fixes: · 1d002ae1
      emarsden authored
        - (fresh-line (make-broadcast-stream)) returns nil instead of t
      
        - STREAM-ELEMENT-TYPE on broadcast-streams should return element-type of
          the last component stream (see CLtS "System Class BROADCAST-STREAM").
          Fix from SBCL.
      
        - WRITE-SEQUENCE handles bit-vectors
      1d002ae1
    • emarsden's avatar
      · cf98504f
      emarsden authored
        - fix WRITE-SEQUENCE for vectors (detected by pfdietz' ANSI compliance
          test suite)
      
          (write-sequence (vector #\a) (make-string-output-stream))
      
          used to fail with
      
          "Type-error in lisp::write-vector-out:  #\a is not of type base-char"
      
          because the WRITE-SEQUENCE code is erroneously assuming that all vectors
          of characters are of type string. There are probably more errors of this
          nature!
      
      
        - fix STREAM-ELEMENT-TYPE for broadcast streams: CLtS specifies that t
          should be returned if there are no component streams, but we used to return
          nil.
      cf98504f
  34. Mar 26, 2004
    • emarsden's avatar
      · 10b760ea
      emarsden authored
       - add a SIMPLE-STREAM-ERROR class, and use it when signaling errors from
         fd-streams DO-OUTPUT (noted by Matthew Danish)
      
       - fix bug in fd-stream error handling (patch from Ole Rohne)
      10b760ea
  35. Jan 20, 2004
    • toy's avatar
      Some ANSI compliance fixes found by Paul Dietz's tests. · 09dc96ef
      toy authored
      o PEEK-CHAR wasn't signaling end-of-file when recursive-p was true.
        o GENERALIZED-PEEKING-MECHANISM needs to handle EOF-DETECTED-FORM
          when the peek-type is NIL.
        o Make PEEK-CHAR with peek-type NIL handle RECURSIVE-P.
      
      o WRITE-LINE didn't like an explicit :END NIL.
      
      o READ-SEQUENCE didn't like reading things into a bit-vector from a
        "good" stream element type.  If the sequence is a bit-vector, use
        the general read-into-vector to read the data.
      
      o READ-SEQUENCE wasn't returning the right value when reading into
        simple-strings with a non-zero :START value.
      09dc96ef
  36. Jan 19, 2004
  37. Oct 24, 2003
    • toy's avatar
      READ-SEQUENCE was returning too soon: · 5b428a0a
      toy authored
      (defvar *s* (open "target:code/scavhook.lisp"))
      (defvar *buf* (make-array 5000 :element-type 'character))
      (read-sequence *buf* *s* :end 3000) => 3000
      (read-sequence *buf* *s* :end 3000) => 1096
      
      or 4096 bytes read, when we should have read 4139 (length of
      scavhook.lisp).
      
      Make READ-INTO-SIMPLE-STRING keep trying to READ-N-BYTES until we have
      enough bytes or until READ-N-BYTES returns 0 bytes read (in which
      case we return a short read because there's nothing left to read).
      5b428a0a
Loading