- Haskell-Cafe - Haskell.org

Re: [Haskell-cafe] Re: Writing binary files?
by Udo Stenzel 16 Sep '04

16 Sep '04

Glynn Clements <glynn.clements(a)virgin.net> schrieb am 16.09.04 10:46:58: > Gabriel Ebner wrote: > > One more reason to fix the I/O functions to handle encodings and have > > a seperate/underlying binary I/O API. > > The problem is that we also need to fix them to handle *no encoding*. What are you proposing here? Making the breakage even worse by specifying a text based api that uses "no encoding"? Having a seperate byte based api is far better. If you don't know the encoding, all you have is bytes, no text. > Also, binary data and text aren't disjoint. Everything is binary; some > of it is *also* text. No, it isn't. Everything is binary (read: we need a byte based io library), after decoding and only after decoding it becomes text (read: we need explicit support for decoding and probably a convenience layer that looks like the old io library). > > String's are a list of unicode characters, [Word8] is a > > list of bytes. > > And what comes out of (and goes into) most core library functions is > the latter. So System.Directory needs to be specified in terms of bytes, too. Looks like a clean solution to me. Regards, Udo. ________________________________________________________________ Verschicken Sie romantische, coole und witzige Bilder per SMS! Jetzt neu bei WEB.DE FreeMail: http://freemail.web.de/?mc=021193

3 2

Re: [Haskell-cafe] Re: Writing binary files?
by MR K P SCHUPKE 16 Sep '04

16 Sep '04

>CommandLineToArgvW provides a way to obtain a Unicode Don't forget there are multiple encodings for unicode: UTF-8, UTF-16, UTF-32... Keean.

1 0

RE: [Haskell-cafe] Re: Writing binary files?
by Simon Marlow 16 Sep '04

16 Sep '04

On 16 September 2004 10:35, Glynn Clements wrote: > Simon Marlow wrote: > >>> Which is why I'm suggesting changing Char to be a byte, so that we >>> can have the basic, robust API now and wait for the more advanced >>> API, rather than having to wait for a usable API while people sort >>> out all of the issues. >> >> An easier way is just to declare that the existing API assumes a >> Latin-1 encoding consistently. Later we might add a way to let the >> application pick another encoding, or request that the I/O library >> uses the locale encoding. > > But how do you do that without breaking stuff? If the application > changes the encoding to UTF-8 (either explicitly, or by using the > locale's encoding when it happens to be UTF-8), then code such as: > > [filename] <- getArgs > openFile filename ReadMode > > will fail if filename isn't a valid UTF-8 sequence. Similarly for the > other cases where the OS accepts/returns byte strings but the Haskell > interface uses String. And that's the correct behaviour, isn't it? Actually I hadn't really considered filenames, I was just talking about data read & written via the IO library. > I'm less concerned about the handling of streams, as you can > reasonably add a way to change the encoding before any data has been > read or written. I'm more concerned about FilePaths, argv, the > environment etc. Yes, these are interesting issues. Filenames are stored as character strings on some OSs (eg. Windows) and byte strings on others. So the Haskell portable API should probably use String, and do decoding based on the locale (if the programmer asks for it). Argv and the environment - I don't know. Windows CreateProcess() allows these to be UTF-16 strings, but I don't know what encoding/decoding happens between CreateProcess() and what the target process sees in its argv[] (can't be bothered to dig through MSDN right now). I suspect these should be Strings in Haskell too, with appropriate decoding/encoding happening under the hood. Cheers, Simon

3 2

RE: [Haskell-cafe] Re: Writing binary files?
by MR K P SCHUPKE 16 Sep '04

16 Sep '04

>E.g. what happens if you call getDirectoryContents for a directory >which contains filenames which aren't valid in the current encoding? Surely this shows the problem with the idea of a 'current encoding' ... You could be reading files from two remote servers each using different encodings... So you could have read and write raw [Word8] and read and write char, somehting like: readWithEncoder :: ([Word8] -> [Char]) -> IO [Char] writeWithEncoder :: ([Char] -> [Word8]) -> [Char] -> IO () After all, how can you 'normalise' a filename to a standard encoding if you don't have a function to do so. Infact if you encounter a server with an encoding you have no converter for, you have no choice but to treat it as raw bytes? Keean.

2 1

RE: [Haskell-cafe] Re: Writing binary files?
by Simon Marlow 16 Sep '04

16 Sep '04

On 16 September 2004 00:02, Glynn Clements wrote: > Which is why I'm suggesting changing Char to be a byte, so that we can > have the basic, robust API now and wait for the more advanced API, > rather than having to wait for a usable API while people sort out all > of the issues. An easier way is just to declare that the existing API assumes a Latin-1 encoding consistently. Later we might add a way to let the application pick another encoding, or request that the I/O library uses the locale encoding. Existing code continues to work, and there are no conceptual problems (a Char is still Unicode). You have to decide what happens when the programmer tries to output a Char that is out of range for Latin-1. The current behaviour is simply to take the code point mod 0x100, but we could also decide to raise an exception in this case. Cheers, Simon

2 1

Unicoded filenames
by Marcin 'Qrczak' Kowalczyk 15 Sep '04

15 Sep '04

Here is what happens when a language provides only narrow-char API for filenames: -------------------- Start of forwarded message -------------------- Date: Wed, 15 Sep 2004 15:18:00 +0100 From: Peter Jolly <peter(a)jollys.org> To: mattias.waldau(a)abc.se CC: caml-list <caml-list(a)inria.fr> Subject: Re: [Caml-list] How do I create files with UTF-8 file names? Mattias Waldau wrote: > I have a filename as an UTF-8 encoded string. I need to be able to > handle strange chars like accents, Asian chars etc. > > Is there any way to create a file with that name? I only need it on Win32. Windows uses UTF-16 for filenames, but provides a non-Unicode interface for legacy applications; the standard open() function that OCaml's open_out wraps appears to use the legacy interface. The precise codepage this uses is system-dependent, and AFAIK there's no way for a program to determine what it is without calling out to the Win32 API, but you can be pretty sure it won't be UTF-8. In other words, there is no reliable way to use a filename containing non-ASCII characters with OCaml's standard library. > Or should I solve this problem by talking directly to the Win32-api? This is probably the best solution. A combination of CreateFileW() and MultiByteToWideChar() should do what you want. ------------------- To unsubscribe, mail caml-list-request(a)inria.fr Archives: http://caml.inria.fr Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/ Beginner's list: http://groups.yahoo.com/group/ocaml_beginners -------------------- End of forwarded message -------------------- -- __("< Marcin Kowalczyk \__/ qrczak(a)knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/

2 1

Layered I/O
by oleg＠pobox.com 15 Sep '04

15 Sep '04

Binary i/o is not specifically a Haskell problem. Other programming systems, for example, Scheme, have been struggling with the same issues. Scheme Binary I/O proposal may therefore be of some interest in this group. http://srfi.schemers.org/srfi-56/ It is deliberately made to be the easiest to implement and the least controversial. There exist more ambitious proposals. The key feature is a layered, or stackable i/o. Enclosed is a justification message that I wrote two years ago for Haskell-Cafe, but somehow did not post. An early draft of the ambitious proposal, cast in the context of Scheme, is available here: http://pobox.com/~oleg/ftp/Scheme/io.txt More polished drafts exist, and even a prototype implementation. Unfortunately, once it became clear that the ideas are working out, the motivation fizzled. The discussion of i18n i/o highlighted the need for general overlay streams. We should be able to place a processing layer onto a handle -- and to peel it off and place another one. The layers can do character encoding, subranging (limiting the stream to the specified number of basic units), base64 and other decoding, signature collecting and verification, etc. Processing of a response from a web server illustrates a genuine need for such overlayed processing. Suppose we have established a connection to a web server, sent a GET or POST request and are now reading the response. It starts as follows: HTTP/1.1 200 Have it Content-type: text/plain; charset=iso-2022-jp Content-length: 12345 Date: Tuesday, August 13, 2002 <empty-line> To read the response line and the content headers, our stream must be in a ASCII, Latin-1 or UTF-8 encoding (regardless of the current locale). The body of the message is encoded in iso-2022-jp. This encoding may have nothing to do with the current locale. Furthermore, many character encodings cannot be reliably detected automatically. Therefore, after we read the headers we must forcibly set our stream to use the iso-2022-jp encoding. ISO-2022-JP is the encoding for Japanese characters [http://www.faqs.org/rfcs/rfc1554.html] It is a variable-length stateful encoding: the start of the specifically Japanese encoding is indicated by \e$B. After that, the reader should read _two octets_ from the input stream (and pass them to the application as they are). The server has indicated that it was sending 12345 _octets_ of data. We cannot tell offhand how many _characters_ of data we will read, because of the variable-length encoding. However, we must not even attempt to read the 12346-th octet: HTTP/1.1 connections are, in general, persistent, and we must not read more data than were sent. Otherwise, we deadlock. Therefore, our stream must be able to give us Japanese characters and still must count octets. The HTTP stream will not, in general, give EOF condition at the end of data. That is not the only complication. Suppose the web server replied: Content-type: text/plain; charset=iso-2022-jp Content-transfer-encoding: chunked We should expect to read a sequence of chunks of the format: <length> CRLF <body> CRLF where <length> is a hexadecimal number, and <body> is encoded as indicated in the Content-type. Therefore, after we read the header, we should keep our stream in the ASCII mode to read the <length> field. After that, we should switch the encoding into ISO-2022-JP. After we have consumed <length> octets, we should switch the stream back to ASCII, verify the trailing CRLF, and read the <length> of the next chunk. The ISO-2022-JP encoding is stateful and wide (a character is formed by several octets). It may well happen that a wide character is split between two chunks: one octet of a character will be in one chunk and the other octets in following chunk. Therefore, when we switch from the ISO-2022-JP encoding to ASCII and back, we must preserve the state of the encoding. This is not the end of the story however. A web server may send us a multi-part reply: a multi-part MIME entity made of several MIME entities, each with its own encoding and transfer modes. Neither of these encoding have anything to do with the current locale. Therefore, we may need to switch encodings back and forth quite a few times. Decoding of such complex streams becomes easier if we can overlay different processing layers on a stream. We start with a TCP handle, overlay an ASCII stream and read the headers, then overlay a stream that reads a specified number of units (and returns EOF when read that many). On the top of the latter we place an ISO-2022-JP decoder. Or we choose a base64-decoder overlayed with a PCS7 signed entity decoder and with a signature verification layer. OpenSSL is one package that offers i/o overlays and stream composition. Overlaying of parsers, encoders and hash accumulators is very common in that particular domain. I have implemented such a facility in two languages, e.g., to overlay an endian stream on the top of a raw stream, then a bit stream and an arithmetic compression stream. In a functional world, Ensemble system [http://citeseer.nj.nec.com/liu99building.html] supports such a stackable i/o. When using overlayed streams, we should remember that all the layers are synchronized. If we did let iso2022_stream = makeiso2022 hFile in body then the raw hFile is still available in the body. Reading the stream with hFile and iso2022_stream indiscriminately will sure wreck havoc. Clean's unique types is an excellent feature to guard against such mistakes. Exactly the same considerations apply if we were using TIFFreader or PNG reader rather than ISO-2022-JP.

2 1

RE: [Haskell-cafe] Writing binary files?
by Simon Marlow 15 Sep '04

15 Sep '04

On 15 September 2004 12:32, ross(a)soi.city.ac.uk wrote: > On Mon, Sep 13, 2004 at 12:01:58PM +0100, Glynn Clements wrote: >> My view is that, right now, we have the worst of both worlds, and >> taking a short step backwards (i.e. narrow the Char type and leave >> the rest alone) is a lot simpler (and more feasible) than the long >> journey towards real I18N. > > This being Haskell, I can't imagine a consensus on a step backwards. > In any case, a Char type distinct from bytes and the rest is the most > valuable part of the current situation. The rest is just libraries, > and the solution to that is to create other libraries. (It's true > that the Prelude is harder to work around, but even that can be done, > as with the new exception interface.) Indeed more than one approach > can proceed concurrently, and that's probably what's going to happen: > > The Right Thing proceeds in stages: > 1. new byte-based libraries > 2. conversions sitting on top of these > 3. the ultimate I18N API > > The Quick Fix: alter the existing implementation to use the > encoding determined by the current locale at the borders. I wish I had some more time to work on this, but I implemented a prototype of an "ultimate" i18n API recently. This is derived from the API that Ben Rudiak-Gould proposed last year. It is in two layers: the InputStream/OutputStream classes provide raw byte I/O, and the TextStream class provides a conversion on top of that. The prototype uses iconv for conversions. You can make Streams from all sorts of things: files, sockets, pipes, and even Haskell arrays. InputStream and OutputStreams are just classes, so you can implement your own. IIRC, I managed to get it working with speed comparable to GHC's current IO library. Here's a tarball that works with GHC 6.2.1 on a Unix platform, just --make to build it: http://www.haskell.org/~simonmar/new-io.tar.gz If anyone would like to pick this up and run with it, I'd be delighted. I'm not likely to get back to in the short term, at least. Cheers, Simon

2 1

Re: [Haskell-cafe] Layered I/O
by MR K P SCHUPKE 15 Sep '04

15 Sep '04

My thoughts on I/O, binary and chars can be summerised: 1) Produce a new WordN based IO library. 2) Character strings cannot be separated from their encodings (ie they must be encoded somehow - even if that encoding is ascii). I would approch this using parameterised phantom types, so for example you would have: data Ascii data Chr a = Char Then the encoding becomes explicit in the type system, but of course functions like equality on types cvan still be expressed generically: charEq :: Chr a -> Chr a -> Bool but of course, you must be comparing the same encodings (enforced by type system)! The phantom type could be used in IO, IE to define a new encodeing: data MyEncoding instance ChrToBinary MyEncoding where chrToBinary = ... instance BinaryToChr MyEncoding where binaryToChr = ... Keean.

1 0

complete documentation like the one from Java or C++
by PRISACARIU Cristian 10 Sep '04

10 Sep '04

Hellow, can you tell me where I can find a more explicit documentation for the Haskell library and maybe for some of the extentions. GHC comes with something but is not as I want. I would like them to tell me for a function what it does not only what arguments it takes and what it returnes. And maybe some hints about how it works if not some examples. I would like something that approch the one from Java or .NET or MSDN. thank you for your time and understanding

5 4