More Unicode nit-picking
I have accidentally noticed another problem in the revised Haskell report (actually the library report, this time). in module Char, toLower and toUpper appear to be under-specified (and indeed, the whole module looks a little suspect) Exactly what is wrong with them I find hard to say, and even harder to say what the cure should be, nevertheless: Firstly: "This module offers only a limited view of the full Unicode character set; the full set of Unicode character attributes is not accessible in this library." I take it this implies that operations in this module apply to the ENTIRE Unicode character set, so: "Function toUpper converts a letter to the corresponding upper-case letter, leaving any other character unchanged. Any Unicode letter which has an upper-case equivalent is transformed. Similarly, toLower converts a letter to the corresponding lower-case letter, leaving any other character unchanged." But this seems to assume there is a one-to-one mapping of upper-case to lower-case equivalent, and vice-versa. Apparently this is not so. (I'm only going on hearsay - something I saw on the xml-dev list yesterday - I haven't checked any of this). It seems that whilst the Unicode database's definitions of whether or not a character is upper/lower/title case are normative, the mappings from upper to lower case are only suggestive. This is because it depends upon language conventions as to how the mapping is done. In Turkish for instance, I is not the upper-case equivalent of i, and vice-versa (apparently there is a dotted i, and a non-dotted i, and likewise for I). So it would seem that the definitions: "-- Case-changing operations toUpper :: Char -> Char toUpper = primUnicodeToUpper toLower :: Char -> Char toLower = primUnicodeToLower" just beg the question - what should the primUnicodeToLower/ToUpper operations actually do? Should they be locale sensitive? -- Colin Paul Adams Preston Lancashire
----- Original Message ----- From: "Colin Paul Adams" ...
But this seems to assume there is a one-to-one mapping of upper-case to lower-case equivalent, and vice-versa. Apparently this is not so.
True. It's quite tricky. See below.
It seems that whilst the Unicode database's definitions of whether or not a character is upper/lower/title case are normative, the mappings from upper to lower case are only suggestive.
Not anymore. TOGETHER with the case mappings in SpecialCasing.txt (ref. below), case mappings are now (very recently) made normative. Locale (or rather, language) specific exceptions are noted in the SpecialCasing.txt file as well as those cases where multiple characters may be returned from one character that is case mapped. Note also that there are contextual requirements, not only concering language, but also surrounding string: e.g. lowercasing a capital sigma should turn into either a lowercase 'ordinary' sigma (when not at end of word) or a terminal sigma (when at end of word). Handling of i, j and related characters are also non-trivial (and some remaining problems with SpecialCasing.txt will hopefully soon be fixed).
This is because it depends upon language conventions as to how the mapping is done. In Turkish for instance, I is not the upper-case equivalent of i, and vice-versa (apparently there is a dotted i, and a non-dotted i, and likewise for I).
There's more to it than that. See UTR 21 (http://www.unicode.org/reports/tr21/), and in particular the UCD (Unicode Character Database) file SpecialCasing.txt (http://www.unicode.org/Public/UNIDATA/SpecialCasing.txt). Kind regards /kent k
participants (2)
-
Colin Paul Adams -
Kent Karlsson