Hello, there was some discussion about Unicode and the Char type some time ago. At the moment I'm writing some Haskell code dealing with XML. The problem is that there seems to be no consensus concerning Char so that it is difficult for me to deal with the XML unicode issues appropriately. Is there any option that is very likely to get into the Haskell report? According to my memory the following were more or less propagated: (1) range '\x0000' to '\xFFFF'; UTF-16 is used (2) range '\x000000' to '\x10FFFF'; chars denote codepoints (3) range '\x00000000' to '\x7FFFFFFF'; chars denote codepoints GHC 5 seems to implement the second variant; Hugs still uses the poor range of '\x00' to '\xFF'. What does nhc98 do? My opinion is that using (1) is very, very bad. The name "Char" suggests that values of the type are characters (glyphs). Well, even when using (2) or (3) Char does not denote characters but codepoints, but this is closer to denoting chars than (1). And memory usage shouldn't be an issue - a concept beeing much better is IMO worth the higher memory use. I prefer (3) over (2) because there is the possibility of expansion of the Unicode character set in the future. Another solution would be to specify that the upper bound of Char is initially '\x10FFFF' and shall be adapted as Unicode evolves. Also, as already said by another person, we should introduce a package dealing with encoding and decoding of character strings to/from octet streams. The encoding used with character I/O must be specified. Any comments? Wolfgang
-----Original Message----- From: haskell-admin@haskell.org [mailto:haskell-admin@haskell.org]On Behalf Of Wolfgang Jeltsch Sent: den 5 januari 2002 13:04 To: The Haskell Mailing List Subject: Unicode again
Hello, there was some discussion about Unicode and the Char type some time ago. At the moment I'm writing some Haskell code dealing with XML. The problem is that there seems to be no consensus concerning Char so that it is difficult for me to deal with the XML unicode issues appropriately. Is there any option that is very likely to get into the Haskell report?
I don't have any opinion on what is likely for Haskell here, but...
According to my memory the following were more or less propagated: (1) range '\x0000' to '\xFFFF'; UTF-16 is used (2) range '\x000000' to '\x10FFFF'; chars denote codepoints (3) range '\x00000000' to '\x7FFFFFFF'; chars denote codepoints
My suggestion here is (2a) range '\x0' to '\xD7FF' union '\xE000' to '\x10FFFF'; chars denote codepoints; the excluded subrange is for "surrogate" codes, they are excluded from UTF-8 and UTF-32, and must occur in proper pairs in UTF-16.
GHC 5 seems to implement the second variant; Hugs still uses the poor range of '\x00' to '\xFF'. What does nhc98 do? My opinion is that using (1) is very, very bad. The name
Not really. Most processes on text must take context into account. Even such a seemingly simple thing as counting what most users think of as characters (or splitting strings at legitimate points) must take context into account. E.g. <a, combining ring above> is one character in the view of the user (though not in the view of the character processing programmer). Everyone that is serious about Unicode and where efficiency is also of concern(!) target UTF-16 (MacOS, Windows, Epoc, Java, Oracle, ...). That does not necessarily mean that Haskell should follow suit.
"Char" suggests that values of the type are characters (glyphs). Well, even
Characters and glyphs are very different concepts. I will not go into detail here, just note that there is not a 1-1 relationship between characters and glyphs even in a single font for many scripts.
when using (2) or (3) Char does not denote characters but codepoints, but this is closer to denoting chars than (1).
Vary marginally. But when looking at individual characters (in the Unicode/10646 sense) UTF-32 is better.
And memory usage shouldn't be an issue - a
For Haskell, I agree.
concept beeing much
s/much/marginally (for strings)/
better is IMO worth the higher memory use. I prefer (3) over (2) because there is the possibility of expansion of the Unicode character set in the future.
Well, it is going in the other direction. With amendment 1 to 10646-1:2000 the limit at 10FFFF is strengthened also in 10646, even though it is not yet as absolute as in Unicode. /kent k
Another solution would be to specify that the upper bound of Char is initially '\x10FFFF' and shall be adapted as Unicode evolves. Also, as already said by another person, we should introduce a package dealing with encoding and decoding of character strings to/from octet streams. The encoding used with character I/O must be specified. Any comments?
Wolfgang
_______________________________________________ Haskell mailing list Haskell@haskell.org http://www.haskell.org/mailman/listinfo/haskell
"Kent Karlsson" <kentk@md.chalmers.se> writes:
Everyone that is serious about Unicode and where efficiency is also of concern(!) target UTF-16 (MacOS, Windows, Epoc, Java, Oracle, ...).
Isn't it fairly common to use 32bit Unicode character types in C? I'm not sure I see the efficiency gain of UTF-16 over UTF-8 or UTF-32, as you still need the multi-unit character management as in UTF-8, while most of the time using more memory. Correct me if I'm wrong, but my impression is that UTF-16 was chosen partly on the assumption that all of Unicode would fit, and I'm not sure it's such an obvious choice today. -kzm -- If I haven't seen further, it is by standing in the footprints of giants
This is getting a bit off-topic for Haskell...
Isn't it fairly common to use 32bit Unicode character types in C?
Yes, in some implementations, but nobody by a few Linux and SunOS programmers use that... (Those systems are far from committed to Unicode.) In some other systems wchar_t is (except for the ASCII part) an unknown (opaque) encoding, literally! Only the system knows the mapping between it and some external encoding. Which renders it completely useless for writing line breaking routines, display routines, collation routines, you name it. Most commonly wchar_t is a 16-bit datatype that holds UTF-16 code units. The Windows APIs use that... Which is agains the C standard, to be nitpicking.
I'm not sure I see the efficiency gain of UTF-16 over UTF-8 or UTF-32, as you still need the multi-unit character management as in UTF-8, while most of the time using more memory.
Correct me if I'm wrong, but my impression is that UTF-16 was chosen partly on the assumption that all of Unicode would fit, and I'm not sure it's such an obvious choice today.
That is not true, but I've explained that before on this list, so I won't do it again; at least not just now. In relation to this: DIN has submitted a request to add (informatively) a new datatype to C: utf16_t. That is based on that some companies (SAP in particular) have found UTF-32 to be too inefficient, even in C programs, and wchar_t cannot conformantly nor portably be used for UTF-16. That does not mean that UTF-32 would not be suitable for Haskell, though. /kent k
participants (3)
-
Kent Karlsson -
Ketil's local user -
Wolfgang Jeltsch