Add support for Unicode 5.2. The normalization and wordbreak tests pass.
code/string.lisp: o In %compose, handle the case where the composite character is outside the BMP and thus needs special handling for our UTF-16 strings. code/unidata.lisp o CKJ Ideograph range has changed in 5.2. o Fix bug in build-composition-table. We were not correctly handling the case where the decomposition of a codepoint was outside the BMP. Special care is needed to handle the UTF-16 strings that we use. o The key for the pairwise composition table are the full codepoints, so we need to shift one by 21 bits instead of 16. tools/build-unidata.lisp o Update minor version to 2. i18n/BidiMirroring.txt i18n/CaseFolding.txt i18n/CompositionExclusions.txt i18n/DerivedNormalizationProps.txt i18n/NameAliases.txt i18n/NormalizationCorrections.txt i18n/SpecialCasing.txt i18n/UnicodeData.txt i18n/WordBreakProperty.txt i18n/tests/NormalizationTest.txt i18n/tests/WordBreakTest.txt o Updated from Unicode 5.2. i18n/unidata.bin o Regenerated from new Unicode 5.2 files.
Showing
- code/string.lisp 15 additions, 5 deletionscode/string.lisp
- code/unidata.lisp 31 additions, 21 deletionscode/unidata.lisp
- i18n/BidiMirroring.txt 5 additions, 5 deletionsi18n/BidiMirroring.txt
- i18n/CaseFolding.txt 10 additions, 4 deletionsi18n/CaseFolding.txt
- i18n/CompositionExclusions.txt 5 additions, 5 deletionsi18n/CompositionExclusions.txt
- i18n/DerivedNormalizationProps.txt 6409 additions, 30 deletionsi18n/DerivedNormalizationProps.txt
- i18n/NameAliases.txt 4 additions, 4 deletionsi18n/NameAliases.txt
- i18n/NormalizationCorrections.txt 3 additions, 3 deletionsi18n/NormalizationCorrections.txt
- i18n/SpecialCasing.txt 9 additions, 9 deletionsi18n/SpecialCasing.txt
- i18n/UnicodeData.txt 2935 additions, 442 deletionsi18n/UnicodeData.txt
- i18n/WordBreakProperty.txt 107 additions, 25 deletionsi18n/WordBreakProperty.txt
- i18n/tests/NormalizationTest.txt 301 additions, 4 deletionsi18n/tests/NormalizationTest.txt
- i18n/tests/WordBreakTest.txt 5 additions, 5 deletionsi18n/tests/WordBreakTest.txt
- i18n/unidata.bin 0 additions, 0 deletionsi18n/unidata.bin
- tools/build-unidata.lisp 2 additions, 2 deletionstools/build-unidata.lisp
Loading
Please register or sign in to comment