Skip to content
Snippets Groups Projects
  1. Jun 29, 2011
    • rtoy's avatar
      Fix Trac #43 again. · 5e7baddc
      rtoy authored
      stream.lisp:
      o Remove the old code that was the first attempt at fixing issue #43.
        This was wrong.
      
      fd-stream.lisp:
      o Fix issue #43.  Even if we have a string-buffer, we need to take
        into account any octets in the in-buffer (if it exists) that have
        not been processed.  This happens if the in-buffer does not have
        enough octets at the end to form a complete character for the given
        encoding.
      o Some debugging prints added, but disabled.
      5e7baddc
  2. May 31, 2011
    • rtoy's avatar
      Ignore shell wildcards when determining next version. · d00265c5
      rtoy authored
      Comment from Paul Foley:
      
        Problem: FD-OPEN calls NEXT-VERSION with the namestring;
        NEXT-VERSION calls EXTRACT-NAME-TYPE-AND-VERSION, which builds a
        "pattern" when it sees globbing characters in the name, and
        NEXT-VERSION promptly crashes when it tries to concatenate strings.
      
        Fix: Bind *IGNORE-WILDCARDS* to T in NEXT-VERSION (in
        fd-stream.lisp)
      d00265c5
  3. Feb 17, 2011
    • rtoy's avatar
      Fix ansi-test failure OPEN.IO.28: · 0e098884
      rtoy authored
      (OPEN PN :DIRECTION :IO :IF-DOES-NOT-EXIST NIL)
      
      no longer signals an error.
      
      code/fd-stream.lisp:
      o Don't try to create a new version if there's no existing file
      
      general-info/release-20c.txt:
      o Update.
      0e098884
  4. Oct 13, 2010
    • rtoy's avatar
      Some changes to replace calls to gettext with _"" or _N"" for things · b22644d4
      rtoy authored
      compiled with and without Unicode.  This is needed so that the pot
      files have the same content for both unicode and non-unicode builds.
      (The _"" and _N"" are handled by the reader, so things that are
      conditionalized out still get processed, unlike using gettext.)
      b22644d4
  5. Oct 12, 2010
    • rtoy's avatar
      Make read-char and read-byte signal errors when given the wrong kind · 35aebeba
      rtoy authored
      of streams.  This is a change from current 20a and 20b behavior which
      didn't signal errors, but matches the behavior for releases 19f and
      earlier.
      
      But allow them to work on binary-text-streams.  This is the same
      behavior as before for binary-text-stream streams.
      
      However, read-sequence no longer allows reading from streams into
      arbitrary objects, unless the stream is a binary-text-stream stream.
      
      code/fd-stream-extfmt.lisp:
      o In %SET-FD-STREAM-EXTERNAL-FORMAT, only update
        fd-stream-in/fd-stream-out if we have a character or
        binary-text-stream stream.
      o Don't update the fd-stream-string-buffer or lisp-stream-in-buffer if
        we have a binary-text-stream because that will mess up how
        fast-read-char and fast-read-byte dispatch to do the right thing for
        binary-text-stream streams.
      
      code/fd-stream.lisp:
      o Set the fd-stream-in and fd-stream-bin slots appropriately depending
        on whether we have a character, binary, or binary-text-stream
        stream.
      o Only create the lisp-stream-in-buffer if we do NOT have a
        binary-text-stream.  (Binary streams didn't use the
        lisp-stream-buffer previously, so no change there.  Character
        streams use the lisp-stream-buffer and/or lisp-string-buffer.)
      o Set the fd-stream-flags appropriately for the kind of stream this
        is.  Checking a fixnum is faster than checking the type of a stream.
      
      code/struct.lisp:
      o Add FLAGS slot to LISP-STREAM so we can tell what kind of stream
        (character, binary, binary-text-stream) we have.
      
      code/sysmacs.lisp:
      o Change FAST-READ-CHAR so that if we have a have a binary or
        binary-text-stream stream, we dispatch to the fast-read-char methods
        to do the right thing, including signaling an error for the wrong
        kind of stream.
      o Change FAST-READ-BYTE so that if we do not have a binary stream, we
        dispatch to the fast-read-char method to do the right thing.
      
      compiler/dump.lisp:
      o With the above changes, we can no longer write characters to a
        binary stream, like a FASL file.  Make the fasl file a
        binary-text-stream so that we can.  (Alternatively, we could create
        the FASL header as a string, convert to octets and dump the octest
        to the file.  This is easier, and should still be fast for writing
        fasls.)
      35aebeba
  6. Sep 15, 2010
  7. Sep 08, 2010
    • rtoy's avatar
      Clean up how (setf stream-external-format) works. This change · 069f884d
      rtoy authored
      requires a cross-compile using the default cross-compile script.
      
      fd-stream.lisp:
      o Remove the SAVED-OC-STATE slot from an FD-STREAM because we don't
        need it anymore since we have the OCTET-COUNT slot.
      
      stream.lisp:
      o Don't need to copy the current OC-STATE to SAVED-OC-STATE.  Remove
        code and update comments.
      
      fd-stream-extfmt.lisp:
      o Use the OCTET-COUNT slot to figure out how many octets have been
        consumed so far to produce the characters that have already been
        read.  Don't need to do the re-conversion anymore, so we don't need
        the SAVED-OC-STATE anymore.
      o Add support for the case where we were using the ISO8859-1 external
        format and are now switching to another external format that
        requires the string-buffer.
      069f884d
  8. Aug 18, 2010
  9. Aug 15, 2010
    • rtoy's avatar
      When decoding-error is T, use a question mark for non-unicode builds · 30ad5edb
      rtoy authored
      instead of the (invalid) Unicode replacement character.
      30ad5edb
    • rtoy's avatar
      Fix file-position bug in trac #36. We add an array to keep track of · d689646f
      rtoy authored
      the octets consumed for each character.  This array is used to figure
      out the file position.  Some tests comparing this scheme indicates a
      very small slowdown of about 1%, so this seems not to hurt.
      
      Use a cross-compile using the 2010-07 snapshot to build this.  (Same
      procedure as used to build the 20b-pre1 release.)
      
      struct.lisp:
      o Add new slot OCTET-COUNT to LISP-STREAM to hold the array of octets
        per character.
      
      extfmts.lisp:
      o Add OCTETS-TO-STRING-COUNTED, which is like OCTETS-TO-STRING, except
        we need an array in which to store the number of octets consumed for
        each character processed.
      
      fd-stream.lisp:
      o Create the octet-count array creating the lisp stream string buffer.
      o In FD-STREAM-FILE-POSITION, use the octet count to count the number
        of octets that have been read but not yet returned to the user.
      
      stream.lisp:
      o Use OCTETS-TO-STRING-COUNTED instead of OCTETS-TO-STRING so we keep
        track of octet length of each character processed.
      d689646f
  10. Aug 04, 2010
  11. Jul 20, 2010
  12. Jul 19, 2010
  13. Jul 05, 2010
  14. Jul 03, 2010
    • rtoy's avatar
      extfmts.lisp: · af5cd1ae
      rtoy authored
      o Update comments for the various slots in DEFINE-EXTERNAL-FORMAT.
      
      fd-stream.lisp:
      o Declare the types for the CHAR-TO-OCTETS-ERROR and
        OCTETS-TO-CHAR-ERROR slots in FD-STREAM.
      o Update docstrings for MAKE-FD-STREAM and OPEN for :DECODING-ERROR
        and :ENCODING-ERROR parameters.
      af5cd1ae
    • rtoy's avatar
      Fix some compiler notes by adding declarations and removing one · 7b95ed2b
      rtoy authored
      variable that was shadowing another.
      7b95ed2b
    • rtoy's avatar
      In EF-FLUSH, we need to handle bare surrogates ourselves for the case · 275f6bf1
      rtoy authored
      where the external format does not have a flush-state method.
      275f6bf1
    • rtoy's avatar
      code/extfmts.lisp: · 6afe1d62
      rtoy authored
      o Add error parameter to flush-state in external format definition so
        we can handle errors when flushing the state to a stream.
      o Add optional error parameter to flush-state macro.
      
      code/fd-stream.lisp:
      o For the case where an external format has flush method, EF-FLUSH was
        not calling it correctly.  Update so the output function actuall
        works.
      o Add error handler to call to flush-state.
      o For the case where an external format does not have a flush method,
        output the state value instead of a replacement character so the
        external format can handle any errors.
      6afe1d62
  15. Jul 02, 2010
    • rtoy's avatar
      code/extfmts.lisp: · eade50bb
      rtoy authored
      o Pass the error handler on for composed external formats.
      
      code/fd-stream.lisp:
      o Forgot to pass the error-handler to char-to-octets in EF-COUT.
      o In MAKE-FD-STREAM slightly change handling of encoding-error and
        decoding-error:
        - If :encoding-error is a character, then the external format will
          use that character whenever an encoding error happens instead of
          the default (internally specified by the external format).
        - If :decoding-error is a character, then the external format will
          use that character whenever an encoding error happens instead of
          the default (internally specified by the external format).  If
          :decoding-error is T, then a cerror is signaled; if continued, the
          Unicode replacement character (#\U+FFFD) is used.
      o Fix bug in OPEN:  The :decoding-error and :encoding-error keyword
        parameter was placed in the &aux section by mistake.
      eade50bb
    • rtoy's avatar
      code/extfmts.lisp: · b1c92748
      rtoy authored
      o The optional error parameter doesn't need to be optional in
        DEFINE-EXTERNAL-FORMAT, EF-STRING-TO-OCTETS, EF-OCTETS-TO-STRING,
        EF-ENCODE and EF-DECODE.
      
      code/fd-stream.lisp:
      o Update comments for char-to-octets-error and octets-to-char-error.
      o Forgot to pass the error handler to char-to-octets in EF-SOUT and
        EF-STRLEN.
      b1c92748
    • rtoy's avatar
      Oops. Remove stray character. · fa932df0
      rtoy authored
      fa932df0
    • rtoy's avatar
      Implement more of the external format error handlers. · f971e392
      rtoy authored
      code/extfmts.lisp
      o Call the error handler for iso8859-1 output.
      o In OCTETS-TO-CODEPOINT and CODEPOINT-TO-OCTETS, call the external
        format with the error argument.
      o In OCTETS-TO-CHAR
        - Call OCTETS-TO-CODEPOINT with the error handler.
        - For all of the error conditions, call the error handler if
          defined.
      o Add error parameter to EF-STRING-TO-OCTETS and EF-ENCODE so we can
        handle errors.  Call CHAR-TO-OCTETS with the error handler.
      o Add error parameter to STRING-TO-OCTETS and use it.
      o Add error parameter to EF-OCTETS-TO-STRING and EF-DECODE so we can
        handle errors.  Call OCTETS-TO-CHAR with the error handler.
      o Add error parameter to OCTETS-TO-STRING and use it.
      o In STRING-ENCODE and STRING-DECODE, call the ef function with the
        error handler.
      o Change STRING-ENCODE to use keyword args instead of optional args.
        Add error parameter and use it.
      
      code/fd-stream-extfmt.lisp:
      o Tell OCTETS-TO-STRING about the error handler stored in the
        fd-stream.
      
      code/fd-stream.lisp:
      o OPEN, MAKE-FD-STREAM, and OPEN-FD-STREAM get DECODING-ERROR and
        ENCODING-ERROR keyword arguments for specifying how to handle
        decoding and encoding errors in external formats.
      
      code/stream.lisp:
      o Make sure the error handler is called in
        FAST-READ-CHAR-STRING-REFILL.
      
      pcl/simple-streams/external-formats/utf-8.lisp:
      o Initial cut at calling the error handler for the various possible
        invalid octet streams for a utf-8 encoding.
      f971e392
  16. Jun 30, 2010
  17. Jun 07, 2010
    • rtoy's avatar
      code/fd-stream.lisp: · c5eff503
      rtoy authored
      o OPEN merges the filename with *DEFAULT-PATHNAME-DEFAULTS*, as
        required by CLHS sec 19.2.3.
      
      code/filesys.lisp:
      o DELETE-FILE, ENSURE-DIRECTORIES-EXIST, and FILE-AUTHOR merge the
        filename with *DEFAULT-PATHNAME-DEFAULTS*, as required by CLHS sec
        19.2.3.
      
      general-info/release-20b.txt:
      o Update.
      c5eff503
  18. Apr 20, 2010
  19. Apr 19, 2010
  20. Mar 19, 2010
  21. Jan 25, 2010
  22. Jan 23, 2010
    • rtoy's avatar
      Oops. Last change to fd-stream doesn't actually compile because no · 2ea523e6
      rtoy authored
      everything is defined yet.  Hence, add dummy
      %SET-FD-STREAM-EXTERNAL-FORMAT and move the real one to
      fd-stream-extfmt.lisp.  This builds.
      
      code/fd-stream.lisp:
      o Always call %SET-FD-STREAM-EXTERNAL-FORMAT, even if
        LISP::*ENABLE-STREAM-BUFFER-P* is NIL.
      
      code/stream.lisp:
      o Move %SET-FD-STREAM-EXTERNAL-FORMAT to fd-stream-extfmt.lisp.
      o Add dummy implementation of %SET-FD-STREAM-EXTERNAL-FORMAT.
      
      code/fd-stream-extfmt.lisp:
      o %SET-FD-STREAM-EXTERNAL-FORMAT moved here.
      2ea523e6
    • rtoy's avatar
      Always set the external format for the stream after calling · fd7d7b38
      rtoy authored
      set-routines so that the stream has the specified external format.
      fd7d7b38
  23. Jan 22, 2010
  24. Oct 18, 2009
    • rtoy's avatar
      Merge changes from unicode-string-buffer-impl-branch which gives · 392d3e59
      rtoy authored
      faster reads on external-formats.  This is done by adding an
      additional buffer to streams so we can convert the entire in-buffer
      into characters all at once.
      
      To build this change, you need to do a cross-compile using
      boot-2009-10-1-cross.lisp.  Using that build, do a normal build with
      these sources.
      
      For a non-unicode build use boot-2009-10-01.lisp with a 20a
      non-unicode build.
      
      code/extfmts.lisp:
      o Add another slot to the extfmts for copying the state.
      o Modify EF-OCTETS-TO-STRING and OCTETS-TO-STRING to support the
        necesssary changes for fast formats.  This is incompatible with the
        previous version because the string is not grown if needed.
      
      code/fd-stream-extfmt.lisp:
      o Set *enable-stream-buffer-p* to T so we have fast streams.
      
      code/fd-stream.lisp:
      o Add new slots to support fast strams.
      o In SET-ROUTINES, initialize the new slots appropriately.
      o Update UNREAD-CHAR to be able to back up in the string buffer to
        unread.
      o Add implementation to copy the state of an external format.
      
      code/stream.lisp:
      o Change %SET-FD-STREAM-EXTERNAL-FORMAT to be able to change formats
        even if we've already converted the buffer with a different format.
        We reconvert the buffer with the old format until we reach the
        current character.  Then the remaining octets are converted using
        the new format and stored in the string buffer.
      o Add FAST-READ-CHAR-STRING-REFILL to refill the string buffer, like
        FAST-READ-CHAR-REFILL does for the octet in-buffer.
      
      code/struct.lisp:
      o Add new slots to hold the string buffer, the current index, and
        length.  These are needed for the fast formats.
      
      code/sysmacs.lisp:
      o Update PREPARE-FOR-FAST-READ-CHAR, DONE-WITH-FAST-READ-CHAR, and
        FAST-READ-CHAR to support the string buffer.
      
      code/string.lisp:
      o Microoptimization of SURROGATEP to reduce the number of branchs.
      
      general-info/release-20b.txt:
      o Update with these changes
      
      pcl/simple-streams/external-formats/utf-16-be.lisp:
      pcl/simple-streams/external-formats/utf-16-le.lisp:
      pcl/simple-streams/external-formats/utf-16.lisp:
      o These formats actually have state, so update them to take a handle
        an initial state.  These are needed if the string buffer ends with a
        leading surrogate and the next string buffer starts with a trailing
        surrogate.  The conversion needs to combine the surrogates together.
      392d3e59
  25. Sep 09, 2009
  26. Aug 26, 2009
    • rtoy's avatar
      Add support for flushing out any state in an external format when · 1325c62e
      rtoy authored
      closing an output stream.  This causes things like
      
      (with-open-file (s "foo" :direction :output :external-format :utf-8)
        (write-char #\u+d800 s))
      
      to output the replacement character instead of creating an empty file.
      
      code/extfmts.lisp:
      o Add new slot to efx structure to hold the function to flush the
        state in an external format.
      o Add accessor for the flush-state slot.
      o Update DEFINE-EXTERNAL-FORMAT to allow specifying the flush
        function.
      o Add macro to call the flush-state function.
      o Added +EF-FLUSH+
      o Use vm::defenum to name the constants instead of the hand-written
        values.
      o Export +REPLACEMENT-CHARACTER-CODE+
      o Document the slots in an efx stucture.
      
      code/fd-stream.lisp:
      o Add ef-flush def-ef-macro to flush the state of an external format
        when closing an output file.  If ef-flush-state is NIL, we just call
        EF-COUT to send out the replacement character.  Otherwise, the
        flush-state function is called to handle it.
      o When closing an output character stream, call ef-flush to flush any
        state before flushing the buffers of the stream.
      o Document the unicode slots in an fd-stream.
      
      code/exports.lisp:
      o Export +REPLACEMENT-CHARACTER-CODE+
      1325c62e
  27. Aug 10, 2009
    • rtoy's avatar
      Fixes from Paul Foley: · 94ed0111
      rtoy authored
      o Standard streams no longer change formats when
        *default-external-format* changes.  Use
        stream:set-system-external-format instead, or (setf
        external-format).
      o char-to-octets properly handles surrogate characters being written.
      o Makes simple-streams work again.
      
      This change needs to be cross-compiled.  2009-07 binaries work for
      cross-compiling using the 19e/boot-2008-05-cross-unicode-*.lisp
      cross-compile script.
      94ed0111
  28. Jun 25, 2009
  29. Jun 24, 2009
  30. Jun 11, 2009
Loading