Document how cmucl handles char and string functions in Unicode
As mentioned in 13.1.10 Documentation of Implementation-Defined Scripts, we need to document the following items (copied from that section):
- Character labels, glyphs, and descriptions. Character labels must be uniquely named using only Latin capital letters A--Z, hyphen (-), and digits 0--9.
- Reader canonicalization. Any mechanisms by which read treats different characters as equivalent must be documented.
- The impact on
char-upcase
,char-downcase
, and the case-sensitive format directives. In particular, for each character with case, whether it is uppercase or lowercase, and which character is its equivalent in the opposite case. - The behavior of the case-insensitive functions
char-equal
,char-not-equal
,char-lessp
,char-greaterp
,char-not-greaterp
, andchar-not-lessp
. - The behavior of any character predicates; in particular, the effects of
alpha-char-p
,lower-case-p
,upper-case-p
,both-case-p
,graphic-char-p
, andalphanumericp
. - The interaction with file I/O, in particular, the supported coded character sets (for example, ISO8859/1-1987) and external encoding schemes supported are documented.
AFAICT, we don't really document any of this. In particular, intern
converts the string to Unicode NFC form before creating the symbol.
And for the case conversion functions, we use case specified by
UnicodeData.txt. This means
(char-downcase (char-upcase <char>))
may return a different
character.
These case conversions also impact strings. In particular
nstring-upcase
may not work for some Unicode characters where the
upcase version of a character takes more than one Unicode code unit.
This also means string-upcase
can end up with a string with a
different length from the original.