30 Sep 2001 22:28:52 +0900, Jens Petersen <petersen@redhat.com> pisze:
16 bits is enough to describe the Basic Multilingual Plane and I think 24 bits all the currently defined extended planes. So I guess the report just refers to the BMP.
In early days the Unicode Consortium was doing everything to confuse peoble about whether Unicode fits into 16 bits. It used to push the view that it's based on 16-bit units, and that pairs of units from the range U+D800..DFFF (called surrogates) can encode a million of extra characters (none of which had a more specific meaning defined at that time). I was told on the Unicode list that it was done because for some people it would be hard to accept an encoding which requires *more* than twice as much storage as 8-bit charsets. 16 bits is "only" twice as much. Unfortunately some companies, like Microsoft and Oracle, believed the "lie of Unicode marketing" and adopted the 16-bit view as the basic internal and external format, ignoring the issue of surrogates. Some time ago the Unicode Consortium slowly began switching to the point of view that abstract characters are denoted by numbers in the range U+0000..10FFFF. Storing them in 16-bit units by expressing characters below U+FFFF directly and representing others as pairs of surrogates is just a way to serialize Unicode to streams of bytes (or 16-bit words), called UTF-16. AFAIK UTF-8 was first present in ISO-10646-1. The ISO standard, although sharing actual assignments of characters to numbers with Unicode, from the beginning viewed character codes as 31-bit numbers, which can be serialized for transmission using for example UTF-8 or UTF-16. Unicode adopted UTF-8 by cutting it at the point of U+10FFFF. It also invented UTF-32 which means to just store characters in 32-bit words (endianness issues are analogous to UTF-16), but is explicitly restricted to characters below U+10FFFF, to avoid confusion with unrestricted 31-bit codes of ISO-10646-1. So now UTF-8, UTF-16 and UTF-32 are treated in parallel by Unicode. The ISO standard is going to match this and limit itself to U+10FFFF too, which in theory should end the problem about the number of characters in these standards. Unicode had to do something with this because it finally began adding characters above U+FFFF, and it would really make no sense to treat UTF-16 as the fundamental view, saying that some codes really don't represent characters but must be used in pairs, since character properties are defined in terms of real characters, not components of surrogate pairs individually. Surrogates are just a hole in the middle of the first 64k of characters, because UTF-16 can't encode them insolated. Unfortunately the 16-bit view is still widespread and there is much confusion. Companies invested money in the 16-bit Unicode and they can't simply replace it with something entirely different, so they actually begin implementing UTF-16. In practice support for surrogates could be almost non-existant in the past, but now there are actual characters allocated there, so it must be done, despite the pain of using a variable-length encoding. There are cases like Oracle which ignored surrogates and misimplemented UTF-8 by treating surrogates like other characters below U+FFFF, yet calling it UTF-8. Now instead of fixing their mistake they added the real UTF-8 under a strange name AL24UTFFSS (I'm not sure if they finally fixed the names) and are trying to push their old version as an official alternative to UTF-8. There is a very strong opposition, but they are still trying. IMHO it would have been better to not invent UTF-16 at all and use UTF-8 in parallel with UTF-32. But Unicode used to promote UTF-16 as the real Unicode, and now it causes so many threads on Unicode list to clear the confusion about the nature of characters above U+FFFF. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
sorry for the me too post, but this has been a major pet peeve of mine for a long time. 16 bit unicode should be gotten rid of, being the worst of both worlds, non backwards compatable with ascii, endianness issues and no constant length encoding.... utf8 externally and utf32 when worknig with individual characters is the way to go. seeing as how the haskell standard is horribly vauge when it comes to character set encodings anyway, I would recommend that we just omit any reference to the bit size of Char, and just say abstractly that each Char represents one unicode character, but the entire range of unicode is not guarenteed to be expressable, which must be true, since haskell 98 implementations can be written now, but unicode can change in the future. The only range guarenteed to be expressable in any representation are the values 0-127 US ASCII (or perhaps latin1) John On Sun, Sep 30, 2001 at 02:29:40PM +0000, Marcin 'Qrczak' Kowalczyk wrote:
IMHO it would have been better to not invent UTF-16 at all and use UTF-8 in parallel with UTF-32. But Unicode used to promote UTF-16 as the real Unicode, and now it causes so many threads on Unicode list to clear the confusion about the nature of characters above U+FFFF.
-- --------------------------------------------------------------------------- John Meacham - California Institute of Technology, Alum. - john@repetae.net ---------------------------------------------------------------------------
On Sunday, 30 September 2001 20:01, John Meacham wrote:
sorry for the me too post, but this has been a major pet peeve of mine for a long time. 16 bit unicode should be gotten rid of, being the worst of both worlds, non backwards compatable with ascii, endianness issues and no constant length encoding.... utf8 externally and utf32 when worknig with individual characters is the way to go.
I totally agree with you.
seeing as how the haskell standard is horribly vauge when it comes to character set encodings anyway, I would recommend that we just omit any reference to the bit size of Char, and just say abstractly that each Char represents one unicode character, but the entire range of unicode is not guarenteed to be expressable, which must be true, since haskell 98 implementations can be written now, but unicode can change in the future. The only range guarenteed to be expressable in any representation are the values 0-127 US ASCII (or perhaps latin1)
This sounds also very good.
John
Wolfgang
----- Original Message ----- From: "Wolfgang Jeltsch" <wolfgang@jeltsch.net> To: "The Haskell Mailing List" <haskell@haskell.org> Sent: Thursday, October 04, 2001 8:47 PM Subject: Re: Unicode support
On Sunday, 30 September 2001 20:01, John Meacham wrote:
sorry for the me too post, but this has been a major pet peeve of mine for a long time. 16 bit unicode should be gotten rid of, being the worst of both worlds, non backwards compatable with ascii, endianness issues and no constant length encoding.... utf8 externally and utf32 when worknig with individual characters is the way to go.
I totally agree with you.
Now, what are your technical arguments for this position? (B.t.w., UTF-16 isn't going to go away, it's very firmly established.)
From what I've seen, those who take the position you seem to prefer, are people not very involved with Unicode and its implementation. Whereas people that are so involved strongly prefer UTF-16.
Note that nearly no string operation of interest (and excepting low level stuff, like buffer sizes, and copying) can be done on a string looking at individual characters only. Just about the only thing that sensibly can be done on isolated characters is property interrogation.You can't do case mapping of a string (involving Greek or Lithuanian text) without being sensitive to the context of each character. And, as somebody already noted, combining characters have to be taken into account. E.g. Å (U+211B (deprecated), or U+00C5) must collate the same as <U+0041,U+030A>, even when not collating them among the A's (U+0041). So it is not surprising that most people involved do not consider UTF-16 a bad idea. The extra complexity is minimal, and further surfaces rarely. Indeed they think UTF-16 is a good idea since the supplementary characters will in most cases occur very rarely, BMP characters are still (relatively) easy to process, and it saves memory space and cache misses when large amounts of text data is processed (e.g. databases). On the other hand, Haskell implementations are probably still rather wasteful when representing strings, and Haskell isn't used to hold large databases, so going to UTF-32 is not a big deal for Haskell, I guess. (Though I don't think that will happen for Java.)
seeing as how the haskell standard is horribly vauge when it comes to character set encodings anyway, I would recommend that we just omit any reference to the bit size of Char, and just say abstractly that each Char represents one unicode character, but the entire range of unicode is not guarenteed to be expressable, which must be true, since haskell 98 implementations can be written now, but unicode can change in the future. The only range guarenteed to be expressable in any representation are the values 0-127 US ASCII (or perhaps latin1)
This sounds also very good.
Why? This is the approach taken by programming languages like C, where the character encoding *at runtime* (both for char and wchar_t) is essentially unknown. This, of course, leads to all sorts of trouble, which some try to mitigate by *suggesting* to have all sorts of locale independent stuff in (POSIX) "locales". Nobody has worked out any sufficiently comprehensive set of data for this though, and nobody ever will, both because it is openended and because nobody is really trying. Furthermore, this is not the approach of Java, Ada, or Haskell. And it is not the approach advocated by people involved with inplementing support for Unicode (and other things related to internationalisation and localisation). Even C is (slowly) leaving that approach, having introduced the __STDC_ISO_10646__ property macro (with it's semantics), and the \uhhhh and \Uhhhhhhhh 'universal character names. Kind regards /kent k
On Sun, Sep 30, 2001 at 11:01:38AM -0700, John Meacham wrote:
seeing as how the haskell standard is horribly vauge when it comes to character set encodings anyway, I would recommend that we just omit any reference to the bit size of Char, and just say abstractly that each Char represents one unicode character, but the entire range of unicode is not guarenteed to be expressable, which must be true, since haskell 98 implementations can be written now, but unicode can change in the future. The only range guarenteed to be expressable in any representation are the values 0-127 US ASCII (or perhaps latin1)
I agree about the vagueness, but I believe the Unicode consortium has explicitly limited itself to 21 bits; if they turn out to have been lying about that (which seems unlikely in this millenium), we can hardly be blamed for believing them. I think all that should be required of implementations is that they support 21 bits. Best, Dylan Thurston
----- Original Message ----- From: "Dylan Thurston" <dpt@math.harvard.edu> To: "John Meacham" <john@repetae.net>; <haskell@haskell.org> Sent: Friday, October 05, 2001 5:47 PM Subject: Re: Unicode support
On Sun, Sep 30, 2001 at 11:01:38AM -0700, John Meacham wrote:
seeing as how the haskell standard is horribly vauge when it comes to character set encodings anyway, I would recommend that we just omit any reference to the bit size of Char, and just say abstractly that each Char represents one unicode character, but the entire range of unicode is not guarenteed to be expressable, which must be true, since haskell 98 implementations can be written now, but unicode can change in the future. The only range guarenteed to be expressable in any representation are the values 0-127 US ASCII (or perhaps latin1)
I agree about the vagueness, but I believe the Unicode consortium has explicitly limited itself to 21 bits; if they turn out to have been
In some sense yes, but not quite. It's better to say that the code space is from 0000 to 10FFFF, then the encoding forms handle the bits.
lying about that (which seems unlikely in this millenium), we can
The guesstimate (originally) of less than half a millon "things" to encode as characters has been stable for over a decade. Even though some try to argue that Unicode had to go from 16-bit to more to be able to handle more characters, that was really known from the beginning. That there was a big bump recently adding 41000 Hàn characters that was collected over a long time and, though some more Hàn are expected, no such big bump. If you're interested, it's gone beyond a guesstimate now, see the roadmap: http://www.evertype.com/standards/iso10646/ucs-roadmap.html (the official version is at the DKUUG site, but the reference is through a cryptic document number). You will see how plane 1 is planned for a number of historical scripts (mostly). Disregarding the private use planes (15 and 16) there is nothing planned for planes 3-14, except for some crap in 14 (what is there is there for political reasons only, DO NOT USE), and that plane 2 may spill over into plane 3. That leaves ten planes (of 64K code positions each) completely empty, with nothing planned for them. Kind regards /kent k
hardly be blamed for believing them. I think all that should be required of implementations is that they support 21 bits.
Best, Dylan Thurston
_______________________________________________ Haskell mailing list Haskell@haskell.org http://www.haskell.org/mailman/listinfo/haskell
"Marcin" == Marcin 'Qrczak' Kowalczyk <qrczak@knm.org.pl> writes:
Marcin> 30 Sep 2001 22:28:52 +0900, Jens Petersen <petersen@redhat.com> pisze: >> 16 bits is enough to describe the Basic Multilingual Plane >> and I think 24 bits all the currently defined extended >> planes. [...] Just for reference... "currently" is the important word here. [Snipped interesting and (hopefully) enlightening stuff about Unicode and marketing] Marcin> IMHO it would have been better to not invent UTF-16 at all Marcin> and use UTF-8 in parallel with UTF-32. But Unicode used to Marcin> promote UTF-16 as the real Unicode, and now it causes so Marcin> many threads on Unicode list to clear the confusion about Marcin> the nature of characters above U+FFFF. IMPPOO is would have been (and would still be) better to simply say "Here's UTF-8, and ISO 10646 is 31 bits, encoded as real 32 bit units or as UTF-8". That's the baseline. None of that "But 21 bits *is* enough". Yeah, like 640K was enough. And countless other examples. I thought we had learned, but I was wrong... I'm especially disheartened to hear that ISO bought into the same crap. Bye, J PS: Yep, this has been an entirely off-topic rant. I simply couldn't resist the urge, the need, to vent a bit. -- Jürgen A. Erhard (juergen.erhard@gmx.net, jae@users.sf.net) My WebHome: http://jerhard.org SPACE: Above And Beyond (http://www.planetx.com/space:aab) I have a firm grip on reality, now I can strangle it.
participants (6)
-
"Jürgen A. Erhard" -
Dylan Thurston -
John Meacham -
Kent Karlsson -
Marcin 'Qrczak' Kowalczyk -
Wolfgang Jeltsch