September 2004 - Haskell-Cafe

RE: [Haskell-cafe] Re: Writing binary files?
by MR K P SCHUPKE 16 Sep '04

16 Sep '04

>In the general case, it needs to be a bit more complex than that, Thats why the functions handled lists not individual characters, I was assuming that each [Word8] -> [Char] represented a valid and complete encoding block... IE at the start of each call it assumes no escapes. All this means is than when reading in chunks you paste those chunks together before conversion, and you can only break outside of escapes. This in my opinion is better behaviour anyway... I don't want some hidden escape state mangling output, just because some earler code generated invalid output. Keean.

2 1

ghc-6-2-1_1.msi MD5 sum
by Sergey Zaharchenko 16 Sep '04

16 Sep '04

Could anyone give me the MD5 sum of http://www.haskell.org/ghc/dist/6.2.1/ghc-6-2-1_1.msi I've been writing in Haskell for FreeBSD, and now I wanted to compile one under windows. The .msi file seems to be broken, and I just wanted to check... -- DoubleF "The more data I punch in this card, the lighter it becomes, and the lower the mailing cost." -- Stan Kelly-Bootle, "The Devil's DP Dictionary"

1 0

Lifting makes lazy
by hjgtuyl＠chello.nl 16 Sep '04

16 Sep '04

L.S., In my enthusiasm to reduce imperative style coding to a minimum, I changed a program to something too lazy to do anything. The following is an extremely simplified version of the program: > import Monad > displayFile1 :: IO (IO ())displayFile1 = liftM putStr contents -- > Displays nothing > where > contents :: IO [Char] > contents = readFile "DisplayFile.lhs" This should display the contents of a file, but nothing appears. The following function should be exactly the same, but this one does display the file: > displayFile2 :: IO () > displayFile2 = do > contents <- readFile "DisplayFile.lhs" > putStr contents My conclusion is, that "putStr" is evaluated strictly, while "liftM putStr" is not. I have the following questions: - Why is this difference? - Is there some method to predict whether my program is sufficiently strict to really do what it is supposed to do? - Did someone design a method to develop programs not too strict and not too lazy? - The manual "A gentle introduction to Haskell" states in section 6.3: "adding strictness flags may lead to hard to find infinite loops or have other unexpected consequences"; I would like to know when these problems arise; are these cases described somewhere? -- Met vriendelijke groet, Herzliche Grüße, Best regards, Henk-Jan van Tuyl -------------------------------------------------------------------- Festina Lente Hasten Slowly Haast U langzaam Eile langsam Skynd dig langsomt Affrettati lentamente Spěchej pomalu Skynda långsamt Desiderius Erasmus --------------------------------------------------------------------

4 3

Re: [Haskell-cafe] Re: Writing binary files?
by Udo Stenzel 16 Sep '04

16 Sep '04

Glynn Clements <glynn.clements(a)virgin.net> schrieb am 16.09.04 10:46:58: > Gabriel Ebner wrote: > > One more reason to fix the I/O functions to handle encodings and have > > a seperate/underlying binary I/O API. > > The problem is that we also need to fix them to handle *no encoding*. What are you proposing here? Making the breakage even worse by specifying a text based api that uses "no encoding"? Having a seperate byte based api is far better. If you don't know the encoding, all you have is bytes, no text. > Also, binary data and text aren't disjoint. Everything is binary; some > of it is *also* text. No, it isn't. Everything is binary (read: we need a byte based io library), after decoding and only after decoding it becomes text (read: we need explicit support for decoding and probably a convenience layer that looks like the old io library). > > String's are a list of unicode characters, [Word8] is a > > list of bytes. > > And what comes out of (and goes into) most core library functions is > the latter. So System.Directory needs to be specified in terms of bytes, too. Looks like a clean solution to me. Regards, Udo. ________________________________________________________________ Verschicken Sie romantische, coole und witzige Bilder per SMS! Jetzt neu bei WEB.DE FreeMail: http://freemail.web.de/?mc=021193

3 2

Re: [Haskell-cafe] Re: Writing binary files?
by MR K P SCHUPKE 16 Sep '04

16 Sep '04

>CommandLineToArgvW provides a way to obtain a Unicode Don't forget there are multiple encodings for unicode: UTF-8, UTF-16, UTF-32... Keean.

1 0

RE: [Haskell-cafe] Re: Writing binary files?
by Simon Marlow 16 Sep '04

16 Sep '04

On 16 September 2004 10:35, Glynn Clements wrote: > Simon Marlow wrote: > >>> Which is why I'm suggesting changing Char to be a byte, so that we >>> can have the basic, robust API now and wait for the more advanced >>> API, rather than having to wait for a usable API while people sort >>> out all of the issues. >> >> An easier way is just to declare that the existing API assumes a >> Latin-1 encoding consistently. Later we might add a way to let the >> application pick another encoding, or request that the I/O library >> uses the locale encoding. > > But how do you do that without breaking stuff? If the application > changes the encoding to UTF-8 (either explicitly, or by using the > locale's encoding when it happens to be UTF-8), then code such as: > > [filename] <- getArgs > openFile filename ReadMode > > will fail if filename isn't a valid UTF-8 sequence. Similarly for the > other cases where the OS accepts/returns byte strings but the Haskell > interface uses String. And that's the correct behaviour, isn't it? Actually I hadn't really considered filenames, I was just talking about data read & written via the IO library. > I'm less concerned about the handling of streams, as you can > reasonably add a way to change the encoding before any data has been > read or written. I'm more concerned about FilePaths, argv, the > environment etc. Yes, these are interesting issues. Filenames are stored as character strings on some OSs (eg. Windows) and byte strings on others. So the Haskell portable API should probably use String, and do decoding based on the locale (if the programmer asks for it). Argv and the environment - I don't know. Windows CreateProcess() allows these to be UTF-16 strings, but I don't know what encoding/decoding happens between CreateProcess() and what the target process sees in its argv[] (can't be bothered to dig through MSDN right now). I suspect these should be Strings in Haskell too, with appropriate decoding/encoding happening under the hood. Cheers, Simon

3 2

RE: [Haskell-cafe] Re: Writing binary files?
by MR K P SCHUPKE 16 Sep '04

16 Sep '04

>E.g. what happens if you call getDirectoryContents for a directory >which contains filenames which aren't valid in the current encoding? Surely this shows the problem with the idea of a 'current encoding' ... You could be reading files from two remote servers each using different encodings... So you could have read and write raw [Word8] and read and write char, somehting like: readWithEncoder :: ([Word8] -> [Char]) -> IO [Char] writeWithEncoder :: ([Char] -> [Word8]) -> [Char] -> IO () After all, how can you 'normalise' a filename to a standard encoding if you don't have a function to do so. Infact if you encounter a server with an encoding you have no converter for, you have no choice but to treat it as raw bytes? Keean.

2 1

RE: [Haskell-cafe] Re: Writing binary files?
by Simon Marlow 16 Sep '04

16 Sep '04

On 16 September 2004 00:02, Glynn Clements wrote: > Which is why I'm suggesting changing Char to be a byte, so that we can > have the basic, robust API now and wait for the more advanced API, > rather than having to wait for a usable API while people sort out all > of the issues. An easier way is just to declare that the existing API assumes a Latin-1 encoding consistently. Later we might add a way to let the application pick another encoding, or request that the I/O library uses the locale encoding. Existing code continues to work, and there are no conceptual problems (a Char is still Unicode). You have to decide what happens when the programmer tries to output a Char that is out of range for Latin-1. The current behaviour is simply to take the code point mod 0x100, but we could also decide to raise an exception in this case. Cheers, Simon

2 1

Unicoded filenames
by Marcin 'Qrczak' Kowalczyk 15 Sep '04

15 Sep '04

Here is what happens when a language provides only narrow-char API for filenames: -------------------- Start of forwarded message -------------------- Date: Wed, 15 Sep 2004 15:18:00 +0100 From: Peter Jolly <peter(a)jollys.org> To: mattias.waldau(a)abc.se CC: caml-list <caml-list(a)inria.fr> Subject: Re: [Caml-list] How do I create files with UTF-8 file names? Mattias Waldau wrote: > I have a filename as an UTF-8 encoded string. I need to be able to > handle strange chars like accents, Asian chars etc. > > Is there any way to create a file with that name? I only need it on Win32. Windows uses UTF-16 for filenames, but provides a non-Unicode interface for legacy applications; the standard open() function that OCaml's open_out wraps appears to use the legacy interface. The precise codepage this uses is system-dependent, and AFAIK there's no way for a program to determine what it is without calling out to the Win32 API, but you can be pretty sure it won't be UTF-8. In other words, there is no reliable way to use a filename containing non-ASCII characters with OCaml's standard library. > Or should I solve this problem by talking directly to the Win32-api? This is probably the best solution. A combination of CreateFileW() and MultiByteToWideChar() should do what you want. ------------------- To unsubscribe, mail caml-list-request(a)inria.fr Archives: http://caml.inria.fr Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/ Beginner's list: http://groups.yahoo.com/group/ocaml_beginners -------------------- End of forwarded message -------------------- -- __("< Marcin Kowalczyk \__/ qrczak(a)knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/

2 1

Layered I/O
by oleg＠pobox.com 15 Sep '04

15 Sep '04

Binary i/o is not specifically a Haskell problem. Other programming systems, for example, Scheme, have been struggling with the same issues. Scheme Binary I/O proposal may therefore be of some interest in this group. http://srfi.schemers.org/srfi-56/ It is deliberately made to be the easiest to implement and the least controversial. There exist more ambitious proposals. The key feature is a layered, or stackable i/o. Enclosed is a justification message that I wrote two years ago for Haskell-Cafe, but somehow did not post. An early draft of the ambitious proposal, cast in the context of Scheme, is available here: http://pobox.com/~oleg/ftp/Scheme/io.txt More polished drafts exist, and even a prototype implementation. Unfortunately, once it became clear that the ideas are working out, the motivation fizzled. The discussion of i18n i/o highlighted the need for general overlay streams. We should be able to place a processing layer onto a handle -- and to peel it off and place another one. The layers can do character encoding, subranging (limiting the stream to the specified number of basic units), base64 and other decoding, signature collecting and verification, etc. Processing of a response from a web server illustrates a genuine need for such overlayed processing. Suppose we have established a connection to a web server, sent a GET or POST request and are now reading the response. It starts as follows: HTTP/1.1 200 Have it Content-type: text/plain; charset=iso-2022-jp Content-length: 12345 Date: Tuesday, August 13, 2002 <empty-line> To read the response line and the content headers, our stream must be in a ASCII, Latin-1 or UTF-8 encoding (regardless of the current locale). The body of the message is encoded in iso-2022-jp. This encoding may have nothing to do with the current locale. Furthermore, many character encodings cannot be reliably detected automatically. Therefore, after we read the headers we must forcibly set our stream to use the iso-2022-jp encoding. ISO-2022-JP is the encoding for Japanese characters [http://www.faqs.org/rfcs/rfc1554.html] It is a variable-length stateful encoding: the start of the specifically Japanese encoding is indicated by \e$B. After that, the reader should read _two octets_ from the input stream (and pass them to the application as they are). The server has indicated that it was sending 12345 _octets_ of data. We cannot tell offhand how many _characters_ of data we will read, because of the variable-length encoding. However, we must not even attempt to read the 12346-th octet: HTTP/1.1 connections are, in general, persistent, and we must not read more data than were sent. Otherwise, we deadlock. Therefore, our stream must be able to give us Japanese characters and still must count octets. The HTTP stream will not, in general, give EOF condition at the end of data. That is not the only complication. Suppose the web server replied: Content-type: text/plain; charset=iso-2022-jp Content-transfer-encoding: chunked We should expect to read a sequence of chunks of the format: <length> CRLF <body> CRLF where <length> is a hexadecimal number, and <body> is encoded as indicated in the Content-type. Therefore, after we read the header, we should keep our stream in the ASCII mode to read the <length> field. After that, we should switch the encoding into ISO-2022-JP. After we have consumed <length> octets, we should switch the stream back to ASCII, verify the trailing CRLF, and read the <length> of the next chunk. The ISO-2022-JP encoding is stateful and wide (a character is formed by several octets). It may well happen that a wide character is split between two chunks: one octet of a character will be in one chunk and the other octets in following chunk. Therefore, when we switch from the ISO-2022-JP encoding to ASCII and back, we must preserve the state of the encoding. This is not the end of the story however. A web server may send us a multi-part reply: a multi-part MIME entity made of several MIME entities, each with its own encoding and transfer modes. Neither of these encoding have anything to do with the current locale. Therefore, we may need to switch encodings back and forth quite a few times. Decoding of such complex streams becomes easier if we can overlay different processing layers on a stream. We start with a TCP handle, overlay an ASCII stream and read the headers, then overlay a stream that reads a specified number of units (and returns EOF when read that many). On the top of the latter we place an ISO-2022-JP decoder. Or we choose a base64-decoder overlayed with a PCS7 signed entity decoder and with a signature verification layer. OpenSSL is one package that offers i/o overlays and stream composition. Overlaying of parsers, encoders and hash accumulators is very common in that particular domain. I have implemented such a facility in two languages, e.g., to overlay an endian stream on the top of a raw stream, then a bit stream and an arithmetic compression stream. In a functional world, Ensemble system [http://citeseer.nj.nec.com/liu99building.html] supports such a stackable i/o. When using overlayed streams, we should remember that all the layers are synchronized. If we did let iso2022_stream = makeiso2022 hFile in body then the raw hFile is still available in the body. Reading the stream with hFile and iso2022_stream indiscriminately will sure wreck havoc. Clean's unique types is an excellent feature to guard against such mistakes. Exactly the same considerations apply if we were using TIFFreader or PNG reader rather than ISO-2022-JP.

2 1