- Haskell-i18n - Haskell.org

Surrogate pairs?
by Sven Moritz Hallberg 20 Aug '02

20 Aug '02

Hi, I just implemented a UTF-8 coder and decoder in Haskell. While reading the Unicode standard I realized what someone had pointed out earlier with respect to code values versus code points: Unicode, while "usually" using 16-bit words, supports "surrogate pairs" to handle all 31 bits of UCS-4. The report says, Char is a 16-bit Unicode value. What's the stance on surrogate pairs? How are we going to support those? My code currently just errors "unsupported" when encountering a surrogate. Regards, Sven Moritz

1 0

Re: [Haskell-i18n] unicode notation \uhhhh implementation
by Ashley Yakeley 16 Aug '02

16 Aug '02

At 2002-08-16 00:46, Ketil Z. Malde wrote: >Personally, I think i'd prefer the identifiers to be renamed to >something more sensible than \uXXXX. Would it be possible to use >an escape character and the *name* of the glyph/symbol instead? Sure, but bear in mind Unicode names for characters are quite long, for instance GREEK SMALL LETTER THETA >I'm also not very happy with backslash for escaping (outside of >strings/char constants), since it's used for lambda (somebody pointed >out this and other problems in a previous mail). Right, but whatever it is it really should be an ASCII character: the point is to allow representation of all identifiers from 7-bit ASCII. -- Ashley Yakeley, Seattle WA

3 4

RE: [Haskell-i18n] unicode notation \uhhhh implementation
by Simon Marlow 16 Aug '02

16 Aug '02

> > I wasn't aware of that paragraph in the report until recently, and > > as far as I know none of the current Haskell implementations > > implement the '\uhhhh' escape sequences. > > HBC implemented Unicode years ago. > > http://www.math.chalmers.se/~augustss/hbc/lexemes.html No, HBC doesn't implement the paragraph of the report that we're talking about. HBC allows the '\uhhhh' escape sequence in characters and string literals, but not in identifiers and other parts of the source. Also, it's not clear to me why you need '\uhhhh' escape sequence in character and string literals at all, since it appears to mean the same thing as '\xhhhh' (the report isn't clear that '\xhhhh' means a "unicode code point", but that seems to be the only reasonable interpretation). > One reason to use this approach would be if there already existed a > preprocessor to do the job - does anyone know of one? > Can't be more than a few lines of Perl. It's quite short in Haskell too: > > convert :: String -> String > convert ('\\':'u':c1:c2:c3:c4:cs) > | isHex c1 && isHex c2 && isHex c3 && isHex c4 > = chr (readHex [c1,c2,c3,c4]) : convert cs > | otherwise -- not clear if this is > = error "Malformed unicode sequence" -- allowed by the spec > convert (c:cs) = c : convert cs > convert [] = [] I meant a preprocessor to take source code in some random encoding and convert it into ASCII with '\uhhhh' escape sequences. If there was such a thing, then we could all use it and save re-implementing N different encodings in each compiler. Cheers, Simon

2 1

RE: [Haskell-i18n] unicode notation \uhhhh implementation
by Sven Moritz Hallberg 16 Aug '02

16 Aug '02

On Fri, 2002-08-16 at 11:01, Simon Marlow wrote: > > > > I wasn't aware of that paragraph in the report until recently, and > > > as far as I know none of the current Haskell implementations > > > implement the '\uhhhh' escape sequences. > > > > HBC implemented Unicode years ago. > > > > http://www.math.chalmers.se/~augustss/hbc/lexemes.html > > No, HBC doesn't implement the paragraph of the report that we're talking about. HBC allows the '\uhhhh' escape sequence in characters and string literals, but not in identifiers and other parts of the source. > > Also, it's not clear to me why you need '\uhhhh' escape sequence in character and string literals at all, since it appears to mean the same thing as '\xhhhh' (the report isn't clear that '\xhhhh' means a "unicode code point", but that seems to be the only reasonable interpretation). You're most probably right, this looks like a misinterpretation on HBC's side to me, too. > > One reason to use this approach would be if there already existed a > > preprocessor to do the job - does anyone know of one? > > > Can't be more than a few lines of Perl. It's quite short in Haskell too: > > > > convert :: String -> String > > convert ('\\':'u':c1:c2:c3:c4:cs) > > | isHex c1 && isHex c2 && isHex c3 && isHex c4 > > = chr (readHex [c1,c2,c3,c4]) : convert cs > > | otherwise -- not clear if this is > > = error "Malformed unicode sequence" -- allowed by the spec > > convert (c:cs) = c : convert cs > > convert [] = [] > > I meant a preprocessor to take source code in some random encoding and convert it into ASCII with '\uhhhh' escape sequences. If there was such a thing, then we could all use it and save re-implementing N different encodings in each compiler. There is GNU recode which knows virtually all kinds of codecs. It's homepage says something about extensibility, so it might be fairly easy to add our own \uhhhh (or whatever we settle on) escaped ASCII to the list of codecs. It could then convert back and forth between this and any other encoding it is aware of. My version reports 281 supported encodings. http://www.gnu.org/software/recode Regards, Sven Moritz

1 0

RE: [Haskell-i18n] unicode notation \uhhhh implementation
by Simon Marlow 16 Aug '02

16 Aug '02

> > I wasn't aware of that paragraph in the report until recently, > > and as far as I know none of the current Haskell implementations > > implement the '\uhhhh' escape sequences. > > > > One reason to use this approach would be if there already existed > > a preprocessor to do the job - does anyone know of one? > > If not, I think the paragraph could be deleted in favour of using > > appropriate encodings for source files (I'd planned to implement > > at least UTF-8 in GHC at some point). > > This means that internally GHC will treat or represent 'Char' as > 32-bit integer ? Yes, in GHC Chars are 32 bits. HsChar is a 32-bit integral type in C. Cheers, Simon

1 0

RE: [Haskell-i18n] unicode notation \uhhhh implementation
by Simon Marlow 16 Aug '02

16 Aug '02

> \u1234 -> u1234 * 5 > "Hey\ \u1234!" > "C:\\udef2\\" > > This is either ambiguous or error-prone. I agree, and the lexical syntax in the report doesn't give any clue as to how the escape sequences are to be interpreted (indeed, it looks like the \uhhhh sequences were an afterthought, which wasn't followed through into the formal syntax). Cheers, Simon

1 0

RE: [Haskell-i18n] unicode notation \uhhhh implementation
by Ashley Yakeley 16 Aug '02

16 Aug '02

At 2002-08-15 10:02, Sven Moritz Hallberg wrote: >What reason would one have to use Unicode characters outside of string >and character literals? The only one I could see would be to have >prettier code. I'm personally itching for real greek letters in my code, >even if only a handful of other people could read the file, ATM. > >But what's the point of it if you can't deal with the characters >yourself, having to use escape sequences. The compiler surely couldn't >care less about it. Well, if we want to go the Java route and allow full Unicode in identifiers, it would allow developer A to export Haskell identifiers with Greek characters from a Greek charset source file, and developer B to use those identifiers from an ASCII source file. -- Ashley Yakeley, Seattle WA

2 1

RE: [Haskell-i18n] unicode notation \uhhhh implementation
by Simon Marlow 16 Aug '02

16 Aug '02

> -----Original Message----- > From: Martin Norbäck [mailto:d95mback@dtek.chalmers.se] > Sent: 15 August 2002 14:40 > To: haskell-i18n(a)haskell.org > Subject: [Haskell-i18n] unicode notation \uhhhh implementation > > > Does anyone know the status of the implementation of unicode escape > sequences \uhhhh as per 2.1 in the Haskell 98 standard? > > When implemented, do the count as one or five characters. > > Or, is UTF-8 (or locale specified encoding) to be used for Haskell > source code? If yes, when? I wasn't aware of that paragraph in the report until recently, and as far as I know none of the current Haskell implementations implement the '\uhhhh' escape sequences. One reason to use this approach would be if there already existed a preprocessor to do the job - does anyone know of one? If not, I think the paragraph could be deleted in favour of using appropriate encodings for source files (I'd planned to implement at least UTF-8 in GHC at some point). Cheers, Simon

3 2

Re: [Haskell-i18n] Some starters for the new list
by Ashley Yakeley 15 Aug '02

15 Aug '02

At 2002-08-14 03:37, Martin Norbäck wrote: >* Unicode library >We need a good Unicode library apart from the standard IO functions that >can be used with specialized IO, in bindings and so on. This should include encoding functions and character properties, at least. If anyone's interested, the build process of my HBase project downloads datafiles from the Unicode web site and converts them into Haskell functions for character properties, that do things such as convert case, decompose, get the General Category etc.. <http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/hbase/Source/HBase/Text/> <http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/hbase/Source/Mak efile?rev=HEAD&content-type=text/plain> -- Ashley Yakeley, Seattle WA

1 0

Some starters for the new list
by Martin Norbäck 15 Aug '02

15 Aug '02

Let's have an open discussion on internationalization issues here. Some issues to discuss. * Unicode in standard IO functions Since Haskell Chars are Unicode, should standard Haskell 98 IO functions be made to obey the locale charset? If the locale charset is iso-8859-1 it will work as before (possibly with some transliteration done, instead of just taking character values modulo 256). See also thread on linux-utf8 mailing list (http://mail.nl.linux.org/linux-utf8/2000-10/msg00139.html) There are no semantics on things like putStr in the library report, there ought to be. * Unicode library We need a good Unicode library apart from the standard IO functions that can be used with specialized IO, in bindings and so on. * Other locale issues Should there be locale dependent show functions (, for decimal separator etc.) * Translation issues What is the best way to make programs translatable? I have some files at http://www.dtek.chalmers.se/~d95mback/gettext/ which could be commented on. I propose that when the Haskell Wiki comes online, we use that to produce draft enhancements to the existing Haskell 98 language report, and perhaps the new Unicode libraries. Regards, Martin -- Martin Norbäck d95mback(a)dtek.chalmers.se Kapplandsgatan 40 +46 (0)708 26 33 60 S-414 78 GÖTEBORG http://www.dtek.chalmers.se/~d95mback/ SWEDEN OpenPGP ID: 3FA8580B

5 6