- Haskell - Haskell.org

Re: UTF-8 library
by Marcin 'Qrczak' Kowalczyk 10 Aug '02

10 Aug '02

Thu, 8 Aug 2002 09:59:12 -0700 (PDT), anatoli <anatoli(a)yahoo.com> pisze: > I'd still rather associate locale with a handle. I agree. http://www.sf.net/projects/qforeign/ contains an experimental character recoding library with a IO module wrapper which associates encodings with Handles. But I don't like my own library (seems ugly and complex). It has rotted somehow, I had unresolved packaging issues witn newest ghcs. -- __("< Marcin Kowalczyk \__/ qrczak(a)knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/

1 0

RE: Text in Haskell: a second proposal
by Ashley Yakeley 10 Aug '02

10 Aug '02

At 2002-08-09 03:26, Simon Marlow wrote: >Why combine I/O and {en,de}coding? Firstly, efficiency. Hmm... surely the encoding functions can be defined efficiently? decodeISO88591 :: [Word8] -> [Char]; encodeISO88591 :: [Char] -> [Word8]; -- uses low octet of codepoint You could surely define them as native functions very efficiently, if necessary. > Secondly, >because it's convenient: if we were to express encodings as stream >transformers, eg: > > decodeUTF8 :: [Word8] -> [Char] > >Then we would have to do all our I/O using lazy streams. You can't >write hGetChar in terms of hGetWord8 using this: you need the non-stream >version which in general looks something like > > decode :: Word8 -> DecodingState > -> (Maybe [Char], DecodingState) > >for UTF-8 you can get away with something simpler, A monadic stream-transformer: decodeStreamUTF8 :: (Monad m) => m Word8 -> m Char; hGetChar h = decodeStreamUTF8 (hGetWord8 h); This works provided each Char corresponds to a contiguous block of Word8s, with no state between them. I think that includes all the standard character encoding schemes. > but AFAIK that's not >true in general. You might want to use compression as an encoding, for >example. Wait a minute... you're not proposing that the existing Char-based IO functions be extended to do compression, are you? If not, I don't see what your point is. -- Ashley Yakeley, Seattle WA

1 0

GHC bug,or Hugs feature?
by Arthur Baars 09 Aug '02

09 Aug '02

One of our students had a program that successfully compiled with Hugs but failed to type check in GHC. After adding some type signatures GHC could compile the program as well. The definitions below are not supposed to make sense, (f and g are just identity functions) but illustrate the situation in the students code: 1) the functions f and g are mutually recursive 2) there is a sort of (indirect) polymorphic recursion - the polymorphic types of f and g are instantiated to (Char->Char) f :: a -> a f a | True = a | g 'a' == 'a' = g a g :: a -> a g a | True = a | f 'a' == 'a' = f a The code above is successfully type-checked by both Hugs and GHC. If we remove either the type signature of f or the type signature of g, however, GHC fails to type check the code. Hugs still infers: f,g :: a->a When removing both type signatures GHC and Hugs both infer f,g::Char->Char, as expected. So this code failed in GHC: f :: a -> a f a | True = a | g 'a' == 'a' = g a g a | True = a | f 'a' == 'a' = f a Is this a bug in GHC or a feature of Hugs? I did not check what NHC and other Haskell compilers say about this. Can someone try? Cheers, Arthur

3 4

Text in Haskell pedantry
by George Russell 09 Aug '02

09 Aug '02

Ashley wrote [quote]No, a file is always a list of octets. Nothing else (ignoring metadata, forks etc.). [/quote] On MVS at least a file is a list of list of octets, because record boundaries are not handled by a "record boundary character" but by other means. There are still more horrible details of MVS access methods no-one here will want to know about. I think it would be more correct to say a file is always a list of C characters, since while this may not be true, at least every system in the foreseeable future is going to make it possible to pretend it is, at least for the sort of files people are likely to want to process using Haskell.

1 0

AlphaBeta (chess) in Haskell
by Mario Lang 09 Aug '02

09 Aug '02

Hello. Does anyone know of a AlphaBeta/Minimax module for haskell, and / or a chess module? I've found several references on the web, but no code. There are little examples in whyfp paper and in some other Haskell related paper, but both are far from complete. Also, I've found a reference to a working mate-problem solver which was used for HAT testing, but also, no code. Maybe someone has something already working which could be used as a basis. My plan is to provide a generic AlphaBeta module which could be used for different kind of two-player games. -- Thanks, Mario

3 2

RE: Text in Haskell: a second proposal
by Simon Marlow 09 Aug '02

09 Aug '02

Here's my take on the Unicode issue. Summary: unless there's a very good reason, I don't think we should decouple encoding/decoding from I/O, at least for the standard I/O library. Firstly, types. We already have all the necessary types: - Char, a Unicode code point - Word8, an octet - CChar, a type representing the C 'char' type The latter two are defined by the FFI addendum. Taking hGetChar as an example: hGetChar :: Handle -> IO Char This combines, IMO, two operations: reading some data from the file, and decoding enough of it to yield a Char. Underneath the hood, the Handle has a particular encoding associated with it. In GHC, currently we have two encodings, ISO8859 (aka binary, but we shouldn't use that term because the I/O library works in terms of Char) and MS-DOS text. We could easily extend the set of encodings to include UTF-8 and others. Seeking only works on Handles with a 1-1 correspondence between handle positions and characters (i.e. in the ISO encoding). Why combine I/O and {en,de}coding? Firstly, efficiency. Secondly, because it's convenient: if we were to express encodings as stream transformers, eg: decodeUTF8 :: [Word8] -> [Char] Then we would have to do all our I/O using lazy streams. You can't write hGetChar in terms of hGetWord8 using this: you need the non-stream version which in general looks something like decode :: Word8 -> DecodingState -> (Maybe [Char], DecodingState) for UTF-8 you can get away with something simpler, but AFAIK that's not true in general. You might want to use compression as an encoding, for example. So in general you need to store not only the DecodingState but also some cached characters between invocations of hGetChar. It's highly unlikely that automatic optimisations will be able to do anything useful with code written using the above interface, but we can write efficient code if the encoder/decoder can work on the I/O buffer directly. There's no reason why we shouldn't provide encoders/decoders as a separate library *as well*, and we should definitely also provide low-level I/O that works with Word8. Cheers, Simon

1 0

Re: Text in Haskell: a second proposal
by Ashley Yakeley 09 Aug '02

09 Aug '02

At 2002-08-09 01:19, Sven Moritz Hallberg wrote: >> Whether or not the old Char-based ones should be deprecated, or whatever, >> I don't know. > >I think any notion of treating the _raw_ contents of a file as Chars >must go, because it is simply incorrect. Right. Certainly we need to come up with _correct_ Word8-based file functions, and separately, text-encoding functions. After that we need to consider what is to be done with the existing "expedient" (and conceptually ugly) Char-based file functions. Should they be deprecated, or should we fix them with a particular encoding scheme such as UTF-8 or ISO 8859-1, or what? What about newline handling? etc. -- Ashley Yakeley, Seattle WA

1 0

Text in Haskell: a second proposal
by Ken Shan 09 Aug '02

09 Aug '02

Thanks to the discussion on this list, I now see that there are not four but five types to be distinguished in text processing: 1. Octets. 2. C "char". 3. Unicode code points. 4. Unicode code values, useful only for UTF-16, which is seldom used. 5. "What handles handle". The new entry in the list is the last one. Wolfgang Jeltsch mentioned "Stream_Element" in Ada 95, which I don't know specifically about but sounds like the same idea. I suggest that the following Haskell types be used for the five items above: 1. Word8 2. CChar 3. CodePoint 4. Word16 5. Char On most machines, Char will be a wrapper around Word8. (This contradicts the present language standard.) Let me elaborate. Files are funny because the information units they contain can be treated as both numbers and characters. Treating these units as numbers, we can convert them to say octets and use them to serialize higher-level data structures such as directed acyclic graphs, bitstrings, and Unicode text. Treating the same units as characters, we can concatenate text like "cat" does, strip out or replace selected characters like "tr" does, or compute the number of units in a stream like "wc -c" does. When we treat a file as containing characters (rather than numbers), we are in effect using a funky character encoding. This encoding is not UTF-8, not ISO-8859-1, and not ASCII. The specifics of this encoding depends on the machine. On mine, it supports 256 characters, mapped to the numbers 0 through 255. Between 0 and 127 is ASCII. From 128 to 255 are 128 additional characters -- let's call them "#128" through "#255". These characters are completely new. For example, "á" is not "#225", just like the integer "225" and the unique complete graph with 225 vertices are not the character "#225", either. Many people will never worry about the characters #128 through #255. This is akin to the fact that the characters "a" through "z" does not concern a program that reads two numbers (in decimal, textual form) from standard input, adds them up, and prints the result to standard out (again in decimal, textual form). As long as the set of characters handled by standard input and standard output includes the decimal digits, the period, and whitespace, the program will work. What do the five Haskell types proposed above mean for the practical programmer? The types in the Haskell IO library will not change; for instance, the only way to read an information unit from a handle is hGetChar :: Handle -> IO Char As I mentioned above, Char under the present proposal is not the type of a Unicode character, but the type of an information unit handled by handles, contra the current language standard. There should, however, be a basic guarantee on how big this information unit is; a reasonable one to make is that it contains at least 8 bits. In other words, ord (chr i) == i for all i such that 0 <= i <= 255. CChar will be a synonym for Char on most systems, just like CInt is a synonym for Int on most systems. I don't know if the sockets library should use Word8 or Char, but it should be one of the two. Now for the tricky issue of converting and defaulting and guessing encodings. In short: Encodings should be handled separately from files. Encoding conversion should be in a library separate from the I/O library. Without involving I/O, I should be able to write a program to answer the question "how many 3-byte sequences are valid UTF-8 text?". We will want stuff in the library like data Encoding text code = Encoding { encode :: [text] -> Maybe [code] , decode :: [code] -> Maybe [text] } utf8 :: Encoding CodePoint Word8 iso88591 :: Encoding CodePoint Word8 as well as char :: Encoding Char Word8 so that UTF-8 conversion from [Char] to [CodePoint] is (>>= decode utf8) . encode char :: [Char] -> Maybe [CodePoint] I am sure many complexities of character encodings are not considered here, but that is in part the point: I would like to see Char in the language standard dissociated from Unicode and return to the more abstract concept of an information unit. -- Edit this signature at http://www.digitas.harvard.edu/cgi-bin/ken/sig http://www.ethnologue.com/

2 1

Re: UTF-8 library
by George Russell 09 Aug '02

09 Aug '02

Axel wrote [snip] > I guess that is a good point, but due to backwards compatibility this is > propably not acceptable: The C interface of the FFI has the string > functions: > peekCString :: CString -> IO String > newCString :: String -> IO CString > > which should really be > > peekCString :: CString -> IO [Word8] > newCString :: [Word8] -> IO CString > > Unless that changes, there is really no point to give the encode and > decode functions that type. [snip] Such a change would be annoying, since I have already used peekCString and newCString quite a lot. (They are a great improvement on what we had before!) Converting CStrings to [Word8] is probably a bad idea anyway, since there is absolutely no reason to assume a C character will be only 8 bits long, and under some implementations it isn't. A better suggestion would be to provide ALTERNATIVE functions which got from CString/CStringLen and friends to [CChar], and make your UTF8 converters go between [CChar] and String. However we should not be forced to do this every time we want to construct a CString from a String (a very common need when calling C functions) so the existing functions should remain with their existing semantics.

4 7

Storable tuples and what is 'alignment'?
by Hal Daume III 09 Aug '02

09 Aug '02

Two things: 1) Why aren't there built in Storable instances for tuples? Writing my own is a pain and in keeping with the 'up to 7tuple' limit, I think built in instances would be nice 2) Could someone tell me what 'alignment' is and if this is the correct definition of a storable tuple: instance (Storable a, Storable b) => Storable (a,b) where sizeOf (a,b) = sizeOf a + sizeOf b alignment (a,b) = alignment a + alignment b peek ptr = do a <- peek (castPtr ptr) b <- peek (castPtr ptr `plusPtr` sizeOf a) return (a,b) poke ptr (a,b) = do poke (castPtr ptr) a poke (castPtr ptr `plusPtr` sizeOf a) b -- Hal Daume III "Computer science is no more about computers | hdaume(a)isi.edu than astronomy is about telescopes." -Dijkstra | www.isi.edu/~hdaume

5 4