August 2002 - Haskell

Re: UTF-8 library
by Ashley Yakeley 11 Aug '02

11 Aug '02

At 2002-08-10 01:21, Marcin 'Qrczak' Kowalczyk wrote: >Perhaps we can assume some widely true facts even if ANSI C doesn't >guarantee that if it makes life easier. For example that a C type >corresponding to Int32 exists at all, and that different pointer >types have the same representation - we already rely on that, don't we? No, we have separate Ptrs and FunctionPtrs IIRC... One of the things that really bothers me about C is the way its unspecifiedness about types can "infect" other languages. For instance, what exactly is a Haskell Int? Java, at least, stands firm, but then platform-independence was one of Java's explicit design priorities. -- Ashley Yakeley, Seattle WA

4 3

Re: UTF-8 library
by Ashley Yakeley 10 Aug '02

10 Aug '02

At 2002-08-08 02:28, Manuel M T Chakravarty wrote: >ANSI C guarantees that char is 1 byte (more precisely that >"sizeof (char)" == 1). That's also what the C++ ARM says (which I have to hand). Unfortunately, "a byte is undefined by the language except in terms of sizeof; sizeof(char) is 1." [sec. 5.3.2] Maybe ANSI C is different? -- Ashley Yakeley, Seattle WA

12 17

Re: Yet more text pedantry
by Marcin 'Qrczak' Kowalczyk 10 Aug '02

10 Aug '02

Fri, 09 Aug 2002 15:24:55 +0200, George Russell <ger(a)tzi.de> pisze: > but the fact is that the standard access functions return > characters*, and on Solaris the default representation of > a characters is as a signed quantity. Only because of a messy history. No need to transfer the silly concept of signed and unsigned characters into Haskell, Haskell doesn't need to sit on top of C - C is just a language. IMHO the best way to view binary file data is Word8. Network protocols and file formats are expressed in terms of octets, so programs which want to cooperate with the rest of the world must be somehow able to read/write them anyway. -- __("< Marcin Kowalczyk \__/ qrczak(a)knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/

1 0

Re: UTF-8 library
by Ashley Yakeley 10 Aug '02

10 Aug '02

At 2002-08-10 03:03, anatoli wrote: >--- Sven Moritz Hallberg <pesco(a)gmx.de> wrote: >> I argue _strongly_ against associating some sort of locale state with >> handles. >> >> 1) In agreement with Ashley's statements, file IO should use octets, >> because that's what's in a file. > >By the same token, we should handle CR/LF/CR-LF/LF-CR mess by hand. >(Files don't have lines in them, they are just sequences of octets.) Correct. Exactly what kind of newline do you want in your file? >I prefer somewhat higher-level view of files. Well, that's what encoding functions are for. You can take higher-level views of your octets as text, images, XML-structures, experimental datasets, whatever. What's so special about text that the functionality should be bound _right into the API_? >> 2) If you need to decode those octets to characters, or vice-versa, >> compose a (de)serialization function before it. > >I *always* need that. (Except for binary IO). You *always* need that. (Except when you don't). The term of "binary" is quite misleading. It suggests a particular file type, but it's actually used to mean "something other than ASCII-compatible text". One might as well have a word that means "something other than a JPEG image". ... >A "Word8 stream" can be either Handle (Word8Handle?) or [Word8]. We can >transform >[Word8] to [Char], but not Word8Handle to CharHandle. I argue that the latter >is needed as well. Well, it should be a utility library built on top of the real Word8-based functions: data TextHandle = MkTextHandle Handle TextEncoding; etc. -- Ashley Yakeley, Seattle WA

2 1

Re: Text in Haskell: a second proposal
by Ashley Yakeley 10 Aug '02

10 Aug '02

At 2002-08-08 23:10, Ken Shan wrote: > 1. Octets. > 2. C "char". > 3. Unicode code points. > 4. Unicode code values, useful only for UTF-16, which is seldom used. > 5. "What handles handle". ... >I suggest that the following Haskell types be used for the five items >above: > > 1. Word8 > 2. CChar > 3. CodePoint > 4. Word16 > 5. Char I disagree, they should be: 1. Word8 2. CChar 3. Char 4. Word16 5. Word8 >Let me elaborate. Files are funny because the information units they >contain can be treated as both numbers and characters. No, a file is always a list of octets. Nothing else (ignoring metadata, forks etc.). Of course, you can interpret those octets as text using "ASCII" or "UTF-8" or whatever, equally, you can interpret those octets as an image using "PNG", "JPEG" etc. But those are secondary transformations, separate from the business of reading from and writing to a file. We should have Word8-based interfaces to file and network handles. Whether or not the old Char-based ones should be deprecated, or whatever, I don't know. As for Unicode codepoints, if there's to be an internationalisation effort for Haskell, the type of character literals, Char, should be fixed as the type for Unicode codepoints, much as it already is in GHC. -- Ashley Yakeley, Seattle WA

3 2

Re: Text in Haskell: A PROPOSAL
by Marcin 'Qrczak' Kowalczyk 10 Aug '02

10 Aug '02

Thu, 8 Aug 2002 03:16:09 -0700, Ashley Yakeley <ashley(a)semantic.org> pisze: >> With, perhaps, UTF-8 as a reasonable default? > > Perhaps it should _always_ be UTF-8? My files are not in UTF-8, so reading them as UTF-8 is wrong. Files are in the locale encoding unless the programmer explicitly specifies otherwise. This is what the locale is for. Filenames are also in the current locale (in my case ISO-8859-2). When Linux has enough tools which deal with UTF-8 (text editors, shells etc.), I might someday switch to UTF-8. Rename files with ISO-8859-2 filenames to UTF-8 on reiserfs partitions, mount the FAT partition as UTF-8 instead of ISO-8859-2 etc., and start using UTF-8 in file contents, and change the locale so gettextized error messages from programs come as UTF-8 instead of ISO-8859-2. It haven't happened yet. Please don't decide for me that I should use UTF-8. I'm not ready yet. 90% of non-ASCII texts I deal with are in ISO-8859-2 and this default is described by the locale. -- __("< Marcin Kowalczyk \__/ qrczak(a)knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/

1 0

Re: UTF-8 library
by Marcin 'Qrczak' Kowalczyk 10 Aug '02

10 Aug '02

Sat, 10 Aug 2002 01:31:51 -0700, Ashley Yakeley <ashley(a)semantic.org> pisze: >>that different pointer >>types have the same representation - we already rely on that, don't we? > > No, we have separate Ptrs and FunctionPtrs IIRC... Yes, but I mean the possibility that Ptr Word8 looks differently than Ptr Word32. > One of the things that really bothers me about C is the way its > unspecifiedness about types can "infect" other languages. For > instance, what exactly is a Haskell Int? It's unrelated to C. Int must have at least 30 bits. It's reasonable to make it 32 bits on IA-32 and 64 bits on IA-64. No need to exactly specify its range because we have Integer and Int8/16/32/64 if needed, and Int is the type appropriate for measuring array lengths for example (we won't have gigabyte arrays in IA-32 but might have them on IA-64). -- __("< Marcin Kowalczyk \__/ qrczak(a)knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/

1 0

Re: UTF-8 library
by Marcin 'Qrczak' Kowalczyk 10 Aug '02

10 Aug '02

09 Aug 2002 10:17:21 +0200, Sven Moritz Hallberg <pesco(a)gmx.de> pisze: > I argue _strongly_ against associating some sort of locale state with > handles. > > 1) In agreement with Ashley's statements, file IO should use octets, > because that's what's in a file. So it would imply two types raw Handles for binary data and wrapped text Handles for strings. You don't want to force users to explicitly perform the conversion over and over I hope? Wrappers would look as IO now. But I don't see much point in separating those Handles. And it would be more efficient if conversion could be done internally on the C side (locale-dependent encoding functions are available from C) coupled with Handle buffers, rather than going through a pure Haskell interface in the middle. -- __("< Marcin Kowalczyk \__/ qrczak(a)knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/

1 0

Re: UTF-8 library
by Marcin 'Qrczak' Kowalczyk 10 Aug '02

10 Aug '02

Thu, 8 Aug 2002 09:59:12 -0700 (PDT), anatoli <anatoli(a)yahoo.com> pisze: > I'd still rather associate locale with a handle. I agree. http://www.sf.net/projects/qforeign/ contains an experimental character recoding library with a IO module wrapper which associates encodings with Handles. But I don't like my own library (seems ugly and complex). It has rotted somehow, I had unresolved packaging issues witn newest ghcs. -- __("< Marcin Kowalczyk \__/ qrczak(a)knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/

1 0

RE: Text in Haskell: a second proposal
by Ashley Yakeley 10 Aug '02

10 Aug '02

At 2002-08-09 03:26, Simon Marlow wrote: >Why combine I/O and {en,de}coding? Firstly, efficiency. Hmm... surely the encoding functions can be defined efficiently? decodeISO88591 :: [Word8] -> [Char]; encodeISO88591 :: [Char] -> [Word8]; -- uses low octet of codepoint You could surely define them as native functions very efficiently, if necessary. > Secondly, >because it's convenient: if we were to express encodings as stream >transformers, eg: > > decodeUTF8 :: [Word8] -> [Char] > >Then we would have to do all our I/O using lazy streams. You can't >write hGetChar in terms of hGetWord8 using this: you need the non-stream >version which in general looks something like > > decode :: Word8 -> DecodingState > -> (Maybe [Char], DecodingState) > >for UTF-8 you can get away with something simpler, A monadic stream-transformer: decodeStreamUTF8 :: (Monad m) => m Word8 -> m Char; hGetChar h = decodeStreamUTF8 (hGetWord8 h); This works provided each Char corresponds to a contiguous block of Word8s, with no state between them. I think that includes all the standard character encoding schemes. > but AFAIK that's not >true in general. You might want to use compression as an encoding, for >example. Wait a minute... you're not proposing that the existing Char-based IO functions be extended to do compression, are you? If not, I don't see what your point is. -- Ashley Yakeley, Seattle WA

1 0