August 2002 - Haskell-i18n

RE: [Haskell-i18n] Unicode in source
by Simon Marlow 21 Aug '02

21 Aug '02

> Apparently, this isn't quite supported by GHC: > > Prelude> map Char.ord "\74\749\7490" > [74,237,66] > > which is, of course, the values modulo 256. I think you've found a bug. It works with a single character: Prelude> Char.ord '\xffff' 65535 but not with a string. Thanks for the report :) > Anyway, if the report is corrected to not limit us to 16 bits, this at > least gives us enough mechanism to use Unicode in string and > character constants. > > What about using it in identifiers? I suggest the following formats: > > #hhhh > and ##hhhhhhhh > > for Unicode characters, with the first form being applicable to code > points below 64K, and the second to all of Unicode. There are several problems with using this kind of encoding in source files, as pointed out by Sven Moritz Hallberg (indentation, syntax ambiguities, etc.), so I'd prefer to stick to standard encodings such as UTF-8 for source files. At least that way you'll be able to get an editor that will display the file as it is indented to be. (aside: aren't there problems with Unicode not being a fixed-width character set? Some characters are expected to combine with others to form a glyph, there are multiple versions of some characters with different widths, there are several widths of space, etc.) Cheers, Simon

2 1

Re: [Haskell-i18n] Surrogate pairs?
by Ashley Yakeley 21 Aug '02

21 Aug '02

At 2002-08-20 04:34, Sven Moritz Hallberg wrote: >I see. I find it pretty inconvenient to read the incremental changes in >the different Unicode revisions. Me too. I have the big blue book (Unicode Standard 3.0), but I have to look at the updates for 3.1 and 3.2. >I've not been able to find the exact >place where they clarify the situation with surrogate pairs. I suppose >what they were is now only a facet of UTF-16, is that correct? I believe so. >Anyway, as you put it, I take it that there should never be a character >composed of two Chars. That's not quite correct. Every code point is exactly one Char, but some characters may be composed of more than one code point. For instance, 'á' might be represented as \#00E1 [LATIN SMALL LETTER A WITH ACUTE] or \#0061 [LATIN SMALL LETTER A] + \#0301 [COMBINING ACUTE ACCENT] > The wording in the report about 16 bits will go, >and the Int representation of Char uses Unicode scalar values. Currently GHC restricts Chars to [0,0x10FFFF], for instance: Prelude> toEnum 0x0061 :: Char 'a' Prelude> toEnum 0x10FFFF :: Char '\1114111' Prelude> toEnum 0x110000 :: Char *** Exception: Prelude.chr: bad argument Prelude> I think this is correct behaviour. -- Ashley Yakeley, Seattle WA

3 2

Re: [Haskell-i18n] Surrogate pairs?
by Ashley Yakeley 21 Aug '02

21 Aug '02

At 2002-08-21 00:17, Ketil Z. Malde wrote: >> \#00E1 [LATIN SMALL LETTER A WITH ACUTE] > >> or > >> \#0061 [LATIN SMALL LETTER A] + \#0301 [COMBINING ACUTE ACCENT] > >I guess they must be treated the same, too? That is, the length of >the strings should be the same, they should compare equal, etc etc. In my opinion no. As far as String is concerned, since it is simply [Char], it should be considered as simply a list of codepoints without further interpretation. So 'length' and its instance for Eq should be the same as for any other list. >Or is it an alternative to just ignore the issue, and simply think of >the latter as two characters? Consider the latter as two codepoints, and don't worry about characters. There should be separate functions for doing such things as decomposition and equivalence. -- Ashley Yakeley, Seattle WA

2 1

Re: [Haskell-i18n] Surrogate pairs?
by Ashley Yakeley 20 Aug '02

20 Aug '02

At 2002-08-19 17:06, Sven Moritz Hallberg wrote: >I just implemented a UTF-8 coder and decoder in Haskell. While reading >the Unicode standard I realized what someone had pointed out earlier >with respect to code values versus code points: Unicode, while "usually" >using 16-bit words, supports "surrogate pairs" to handle all 31 bits of >UCS-4. > >The report says, Char is a 16-bit Unicode value. Right, sec. 6.1.2. But this should change. A Char should allow (and only allow) values in the range [0,0x110000 - 1]. These are _Unicode scalar values_ as defined in the standard sec. 3.7, D28. Unicode scalar values are also known as "code positions" or "code points". The current version of GHC does precisely this. I don't think UCS-4 is used anymore, all character assignments are to code points, 0 to 0x10FFFF. > What's the stance on surrogate pairs? > How are we going to support those? My code currently >just errors "unsupported" when encountering a surrogate. I think we should be working to the latest version of Unicode, 3.2.0. <http://www.unicode.org/unicode/reports/tr28/> If your UTF-8 decoder comes across a sequence apparently representing a codepoint in the range [0xD800,0xDFFF], it should consider it "ill-formed". This is a new thing in 3.2. If Chars are code points rather than 16-bit code values, then when your UTF-8 decoder comes across a sequence representing a codepoint in the range [0x10000,0x10FFFF], it should represent it as a single Char, not as a surrogate pair of Chars. Surrogate pairs are for UTF-16, AFAIK they're not supposed to exist as code points. -- Ashley Yakeley, Seattle WA

2 1

Surrogate pairs?
by Sven Moritz Hallberg 20 Aug '02

20 Aug '02

Hi, I just implemented a UTF-8 coder and decoder in Haskell. While reading the Unicode standard I realized what someone had pointed out earlier with respect to code values versus code points: Unicode, while "usually" using 16-bit words, supports "surrogate pairs" to handle all 31 bits of UCS-4. The report says, Char is a 16-bit Unicode value. What's the stance on surrogate pairs? How are we going to support those? My code currently just errors "unsupported" when encountering a surrogate. Regards, Sven Moritz

1 0

Re: [Haskell-i18n] unicode notation \uhhhh implementation
by Ashley Yakeley 16 Aug '02

16 Aug '02

At 2002-08-16 00:46, Ketil Z. Malde wrote: >Personally, I think i'd prefer the identifiers to be renamed to >something more sensible than \uXXXX. Would it be possible to use >an escape character and the *name* of the glyph/symbol instead? Sure, but bear in mind Unicode names for characters are quite long, for instance GREEK SMALL LETTER THETA >I'm also not very happy with backslash for escaping (outside of >strings/char constants), since it's used for lambda (somebody pointed >out this and other problems in a previous mail). Right, but whatever it is it really should be an ASCII character: the point is to allow representation of all identifiers from 7-bit ASCII. -- Ashley Yakeley, Seattle WA

3 4

RE: [Haskell-i18n] unicode notation \uhhhh implementation
by Simon Marlow 16 Aug '02

16 Aug '02

> > I wasn't aware of that paragraph in the report until recently, and > > as far as I know none of the current Haskell implementations > > implement the '\uhhhh' escape sequences. > > HBC implemented Unicode years ago. > > http://www.math.chalmers.se/~augustss/hbc/lexemes.html No, HBC doesn't implement the paragraph of the report that we're talking about. HBC allows the '\uhhhh' escape sequence in characters and string literals, but not in identifiers and other parts of the source. Also, it's not clear to me why you need '\uhhhh' escape sequence in character and string literals at all, since it appears to mean the same thing as '\xhhhh' (the report isn't clear that '\xhhhh' means a "unicode code point", but that seems to be the only reasonable interpretation). > One reason to use this approach would be if there already existed a > preprocessor to do the job - does anyone know of one? > Can't be more than a few lines of Perl. It's quite short in Haskell too: > > convert :: String -> String > convert ('\\':'u':c1:c2:c3:c4:cs) > | isHex c1 && isHex c2 && isHex c3 && isHex c4 > = chr (readHex [c1,c2,c3,c4]) : convert cs > | otherwise -- not clear if this is > = error "Malformed unicode sequence" -- allowed by the spec > convert (c:cs) = c : convert cs > convert [] = [] I meant a preprocessor to take source code in some random encoding and convert it into ASCII with '\uhhhh' escape sequences. If there was such a thing, then we could all use it and save re-implementing N different encodings in each compiler. Cheers, Simon

2 1

RE: [Haskell-i18n] unicode notation \uhhhh implementation
by Sven Moritz Hallberg 16 Aug '02

16 Aug '02

On Fri, 2002-08-16 at 11:01, Simon Marlow wrote: > > > > I wasn't aware of that paragraph in the report until recently, and > > > as far as I know none of the current Haskell implementations > > > implement the '\uhhhh' escape sequences. > > > > HBC implemented Unicode years ago. > > > > http://www.math.chalmers.se/~augustss/hbc/lexemes.html > > No, HBC doesn't implement the paragraph of the report that we're talking about. HBC allows the '\uhhhh' escape sequence in characters and string literals, but not in identifiers and other parts of the source. > > Also, it's not clear to me why you need '\uhhhh' escape sequence in character and string literals at all, since it appears to mean the same thing as '\xhhhh' (the report isn't clear that '\xhhhh' means a "unicode code point", but that seems to be the only reasonable interpretation). You're most probably right, this looks like a misinterpretation on HBC's side to me, too. > > One reason to use this approach would be if there already existed a > > preprocessor to do the job - does anyone know of one? > > > Can't be more than a few lines of Perl. It's quite short in Haskell too: > > > > convert :: String -> String > > convert ('\\':'u':c1:c2:c3:c4:cs) > > | isHex c1 && isHex c2 && isHex c3 && isHex c4 > > = chr (readHex [c1,c2,c3,c4]) : convert cs > > | otherwise -- not clear if this is > > = error "Malformed unicode sequence" -- allowed by the spec > > convert (c:cs) = c : convert cs > > convert [] = [] > > I meant a preprocessor to take source code in some random encoding and convert it into ASCII with '\uhhhh' escape sequences. If there was such a thing, then we could all use it and save re-implementing N different encodings in each compiler. There is GNU recode which knows virtually all kinds of codecs. It's homepage says something about extensibility, so it might be fairly easy to add our own \uhhhh (or whatever we settle on) escaped ASCII to the list of codecs. It could then convert back and forth between this and any other encoding it is aware of. My version reports 281 supported encodings. http://www.gnu.org/software/recode Regards, Sven Moritz

1 0

RE: [Haskell-i18n] unicode notation \uhhhh implementation
by Simon Marlow 16 Aug '02

16 Aug '02

> > I wasn't aware of that paragraph in the report until recently, > > and as far as I know none of the current Haskell implementations > > implement the '\uhhhh' escape sequences. > > > > One reason to use this approach would be if there already existed > > a preprocessor to do the job - does anyone know of one? > > If not, I think the paragraph could be deleted in favour of using > > appropriate encodings for source files (I'd planned to implement > > at least UTF-8 in GHC at some point). > > This means that internally GHC will treat or represent 'Char' as > 32-bit integer ? Yes, in GHC Chars are 32 bits. HsChar is a 32-bit integral type in C. Cheers, Simon

1 0

RE: [Haskell-i18n] unicode notation \uhhhh implementation
by Simon Marlow 16 Aug '02

16 Aug '02

> \u1234 -> u1234 * 5 > "Hey\ \u1234!" > "C:\\udef2\\" > > This is either ambiguous or error-prone. I agree, and the lexical syntax in the report doesn't give any clue as to how the escape sequences are to be interpreted (indeed, it looks like the \uhhhh sequences were an afterthought, which wasn't followed through into the formal syntax). Cheers, Simon

1 0