August 2002 - Haskell-i18n

RE: [Haskell-i18n] unicode notation \uhhhh implementation
by Ashley Yakeley 16 Aug '02

16 Aug '02

At 2002-08-15 10:02, Sven Moritz Hallberg wrote: >What reason would one have to use Unicode characters outside of string >and character literals? The only one I could see would be to have >prettier code. I'm personally itching for real greek letters in my code, >even if only a handful of other people could read the file, ATM. > >But what's the point of it if you can't deal with the characters >yourself, having to use escape sequences. The compiler surely couldn't >care less about it. Well, if we want to go the Java route and allow full Unicode in identifiers, it would allow developer A to export Haskell identifiers with Greek characters from a Greek charset source file, and developer B to use those identifiers from an ASCII source file. -- Ashley Yakeley, Seattle WA

2 1

RE: [Haskell-i18n] unicode notation \uhhhh implementation
by Simon Marlow 16 Aug '02

16 Aug '02

> -----Original Message----- > From: Martin Norbäck [mailto:d95mback@dtek.chalmers.se] > Sent: 15 August 2002 14:40 > To: haskell-i18n(a)haskell.org > Subject: [Haskell-i18n] unicode notation \uhhhh implementation > > > Does anyone know the status of the implementation of unicode escape > sequences \uhhhh as per 2.1 in the Haskell 98 standard? > > When implemented, do the count as one or five characters. > > Or, is UTF-8 (or locale specified encoding) to be used for Haskell > source code? If yes, when? I wasn't aware of that paragraph in the report until recently, and as far as I know none of the current Haskell implementations implement the '\uhhhh' escape sequences. One reason to use this approach would be if there already existed a preprocessor to do the job - does anyone know of one? If not, I think the paragraph could be deleted in favour of using appropriate encodings for source files (I'd planned to implement at least UTF-8 in GHC at some point). Cheers, Simon

3 2

Re: [Haskell-i18n] Some starters for the new list
by Ashley Yakeley 15 Aug '02

15 Aug '02

At 2002-08-14 03:37, Martin Norbäck wrote: >* Unicode library >We need a good Unicode library apart from the standard IO functions that >can be used with specialized IO, in bindings and so on. This should include encoding functions and character properties, at least. If anyone's interested, the build process of my HBase project downloads datafiles from the Unicode web site and converts them into Haskell functions for character properties, that do things such as convert case, decompose, get the General Category etc.. <http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/hbase/Source/HBase/Text/> <http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/hbase/Source/Mak efile?rev=HEAD&content-type=text/plain> -- Ashley Yakeley, Seattle WA

1 0

Some starters for the new list
by Martin Norbäck 15 Aug '02

15 Aug '02

Let's have an open discussion on internationalization issues here. Some issues to discuss. * Unicode in standard IO functions Since Haskell Chars are Unicode, should standard Haskell 98 IO functions be made to obey the locale charset? If the locale charset is iso-8859-1 it will work as before (possibly with some transliteration done, instead of just taking character values modulo 256). See also thread on linux-utf8 mailing list (http://mail.nl.linux.org/linux-utf8/2000-10/msg00139.html) There are no semantics on things like putStr in the library report, there ought to be. * Unicode library We need a good Unicode library apart from the standard IO functions that can be used with specialized IO, in bindings and so on. * Other locale issues Should there be locale dependent show functions (, for decimal separator etc.) * Translation issues What is the best way to make programs translatable? I have some files at http://www.dtek.chalmers.se/~d95mback/gettext/ which could be commented on. I propose that when the Haskell Wiki comes online, we use that to produce draft enhancements to the existing Haskell 98 language report, and perhaps the new Unicode libraries. Regards, Martin -- Martin Norbäck d95mback(a)dtek.chalmers.se Kapplandsgatan 40 +46 (0)708 26 33 60 S-414 78 GÖTEBORG http://www.dtek.chalmers.se/~d95mback/ SWEDEN OpenPGP ID: 3FA8580B

5 6

Re: [Haskell-i18n] Some starters for the new list
by Marcin 'Qrczak' Kowalczyk 15 Aug '02

15 Aug '02

15 Aug 2002 13:11:46 +0100, Alastair Reid <alastair(a)reid-consulting-uk.ltd.uk> pisze: > data Char = .. -- unicode > data Latin1 = ... -- Latin1 I don't see much use of this. If we want to select encoding dynamically (e.g. mail / news / WWW / irc client which selects it explicitly basing on metadata or user settings), they can't all use their own types. Internal texts will probably be Unicode, externally they are streams of bytes (of Word8). > instance String FrenchString where > fromString (FS "General Protection Fault") = "..." > fromString (FS "File not found") = "..." > ... > fromString (FS _) = ???? Such data is usually read from files, and the language is usually chosen dynamically basing on user settings. Again different languages can't use their own types. The fact that translating a string requires IO will cause trouble. Perhaps we should hide this IO in a function which appears to be pure (uses unsafePerformIO internally and uses the locale from the time the "function" was created, not affected by a setlocale equivalent). -- __("< Marcin Kowalczyk \__/ qrczak(a)knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/

1 0

RE: [Haskell-i18n] unicode notation \uhhhh implementation
by Sven Moritz Hallberg 15 Aug '02

15 Aug '02

On Thu, 2002-08-15 at 16:10, Simon Marlow wrote: > > -----Original Message----- > > From: Martin Norbäck [mailto:d95mback@dtek.chalmers.se] > > Sent: 15 August 2002 14:40 > > To: haskell-i18n(a)haskell.org > > Subject: [Haskell-i18n] unicode notation \uhhhh implementation > > > > > > Does anyone know the status of the implementation of unicode escape > > sequences \uhhhh as per 2.1 in the Haskell 98 standard? > > > > When implemented, do the count as one or five characters. > > > > Or, is UTF-8 (or locale specified encoding) to be used for Haskell > > source code? If yes, when? > > I wasn't aware of that paragraph in the report until recently, and as far as I know none of the current Haskell implementations implement the '\uhhhh' escape sequences. > > One reason to use this approach would be if there already existed a preprocessor to do the job - does anyone know of one? If not, I think the paragraph could be deleted in favour of using appropriate encodings for source files (I'd planned to implement at least UTF-8 in GHC at some point). What reason would one have to use Unicode characters outside of string and character literals? The only one I could see would be to have prettier code. I'm personally itching for real greek letters in my code, even if only a handful of other people could read the file, ATM. But what's the point of it if you can't deal with the characters yourself, having to use escape sequences. The compiler surely couldn't care less about it. One reason to use such a Unicode-to-ASCII conversion would be to enable others whose editors and other tools don't support the encoding to read your code. They'd need a tool to escape your encoding, though. But after that, it might indeed be convenient if the compiler preprocessed the escapes automatically. The only question that remains is: how likely are things like these: \u1234 -> u1234 * 5 "Hey\ \u1234!" "C:\\udef2\\" This is either ambiguous or error-prone. I think I'd personally vote against allowing this in the language. I guess the problem is that the backslash appears in two unrelated "layers" so one can never be quite sure about which layer it was intended for. Of course a good definition can be made. The current behaviour is already well-defined: the above for instance would be a syntax error and a couple of strings with strange contents (Hm, would the first string also be a syntax error?). But what would be the cost for avoiding these things? The first case would harm code-generators. Hm. The second one looks artificial. But number three doesn't appear all that strange and alot of people would stumble over that, not knowing what's up, I'm sure. Having shared the relevant contents of my mind to feed the process, I leave the final decision up to people with more experience. Regards, Sven Moritz

1 0

unicode notation \uhhhh implementation
by Martin Norbäck 15 Aug '02

15 Aug '02

Does anyone know the status of the implementation of unicode escape sequences \uhhhh as per 2.1 in the Haskell 98 standard? When implemented, do the count as one or five characters. Or, is UTF-8 (or locale specified encoding) to be used for Haskell source code? If yes, when? Regards, Martin -- Martin Norbäck d95mback(a)dtek.chalmers.se Kapplandsgatan 40 +46 (0)708 26 33 60 S-414 78 GÖTEBORG http://www.dtek.chalmers.se/~d95mback/ SWEDEN OpenPGP ID: 3FA8580B

1 0

RE: Text in Haskell: a second proposal
by Simon Marlow 15 Aug '02

15 Aug '02

[ moving to haskell-i18n(a)haskell.org ] > For ISO-8859-1 each Char is exactly one Word8, so surely it > would work fine with partial reads? > > decodeCharISO88591 :: Word8 -> Char; > > encodeCharISO88591 :: Char -> Word8; > > decodeISO88591 :: [Word8] -> [Char]; > decodeISO88591 = fmap decodeCharISO88591; > > encodeISO88591 :: [Char] -> [Word8]; > encodeISO88591 = fmap encodeCharISO88591; Sorry, I thought you were just using ISO8859-1 as an example. > >This is better: it doesn't force you to use lazy I/O, and when > >specialised to the IO monad it might get decent performance. The > >problem is that in general I don't think you can assume the lack of > >state. For example: UTF-7 has a state which needs to be retained > >between characters, and UTF-16 and UTF-32 have an endianness > state which > >can be changed by a special sequence at the beginning of the > file. Some > >other encodings have states too. > > But it is possible to do this in Haskell... > > The rule for the many functions in the standard libraries seems to be > "implement as much in Haskell as possible". Why is it any > different with the file APIs? I think we've lost track of the discussion here... I'll try to summarise. I think character encoding/decoding should be built-in to the I/O system. I also think there should be a low-level I/O interface that doesn't do any encoding, and high-level interfaces to the various encodings. Now, you can by all means specify the high-level I/O in terms of the low-level I/O + encodings, but I strongly suspect that implementing it that way will be expensive. Character I/O in Haskell is *already* very slow (see Doug Bagely's language shootout for evidence), and I don't want to add another factor of 2 or more to that. The point is that by building encoding into the I/O interface the implementor gets the opportunity to optimise. Cheers, Simon

1 0

Re: [Haskell-i18n] Some starters for the new list
by Ashley Yakeley 14 Aug '02

14 Aug '02

At 2002-08-14 03:37, Martin Norbäck wrote: >Let's have an open discussion on internationalization issues here. I think you should repost this in 24 hours, to give everyone a chance to subscribe to the list. -- Ashley Yakeley, Seattle WA

1 0