- Haskell-Cafe - Haskell.org

Unicoded filenames
by Marcin 'Qrczak' Kowalczyk 15 Sep '04

15 Sep '04

Here is what happens when a language provides only narrow-char API for filenames: -------------------- Start of forwarded message -------------------- Date: Wed, 15 Sep 2004 15:18:00 +0100 From: Peter Jolly <peter(a)jollys.org> To: mattias.waldau(a)abc.se CC: caml-list <caml-list(a)inria.fr> Subject: Re: [Caml-list] How do I create files with UTF-8 file names? Mattias Waldau wrote: > I have a filename as an UTF-8 encoded string. I need to be able to > handle strange chars like accents, Asian chars etc. > > Is there any way to create a file with that name? I only need it on Win32. Windows uses UTF-16 for filenames, but provides a non-Unicode interface for legacy applications; the standard open() function that OCaml's open_out wraps appears to use the legacy interface. The precise codepage this uses is system-dependent, and AFAIK there's no way for a program to determine what it is without calling out to the Win32 API, but you can be pretty sure it won't be UTF-8. In other words, there is no reliable way to use a filename containing non-ASCII characters with OCaml's standard library. > Or should I solve this problem by talking directly to the Win32-api? This is probably the best solution. A combination of CreateFileW() and MultiByteToWideChar() should do what you want. ------------------- To unsubscribe, mail caml-list-request(a)inria.fr Archives: http://caml.inria.fr Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/ Beginner's list: http://groups.yahoo.com/group/ocaml_beginners -------------------- End of forwarded message -------------------- -- __("< Marcin Kowalczyk \__/ qrczak(a)knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/

2 1

Layered I/O
by oleg＠pobox.com 15 Sep '04

15 Sep '04

Binary i/o is not specifically a Haskell problem. Other programming systems, for example, Scheme, have been struggling with the same issues. Scheme Binary I/O proposal may therefore be of some interest in this group. http://srfi.schemers.org/srfi-56/ It is deliberately made to be the easiest to implement and the least controversial. There exist more ambitious proposals. The key feature is a layered, or stackable i/o. Enclosed is a justification message that I wrote two years ago for Haskell-Cafe, but somehow did not post. An early draft of the ambitious proposal, cast in the context of Scheme, is available here: http://pobox.com/~oleg/ftp/Scheme/io.txt More polished drafts exist, and even a prototype implementation. Unfortunately, once it became clear that the ideas are working out, the motivation fizzled. The discussion of i18n i/o highlighted the need for general overlay streams. We should be able to place a processing layer onto a handle -- and to peel it off and place another one. The layers can do character encoding, subranging (limiting the stream to the specified number of basic units), base64 and other decoding, signature collecting and verification, etc. Processing of a response from a web server illustrates a genuine need for such overlayed processing. Suppose we have established a connection to a web server, sent a GET or POST request and are now reading the response. It starts as follows: HTTP/1.1 200 Have it Content-type: text/plain; charset=iso-2022-jp Content-length: 12345 Date: Tuesday, August 13, 2002 <empty-line> To read the response line and the content headers, our stream must be in a ASCII, Latin-1 or UTF-8 encoding (regardless of the current locale). The body of the message is encoded in iso-2022-jp. This encoding may have nothing to do with the current locale. Furthermore, many character encodings cannot be reliably detected automatically. Therefore, after we read the headers we must forcibly set our stream to use the iso-2022-jp encoding. ISO-2022-JP is the encoding for Japanese characters [http://www.faqs.org/rfcs/rfc1554.html] It is a variable-length stateful encoding: the start of the specifically Japanese encoding is indicated by \e$B. After that, the reader should read _two octets_ from the input stream (and pass them to the application as they are). The server has indicated that it was sending 12345 _octets_ of data. We cannot tell offhand how many _characters_ of data we will read, because of the variable-length encoding. However, we must not even attempt to read the 12346-th octet: HTTP/1.1 connections are, in general, persistent, and we must not read more data than were sent. Otherwise, we deadlock. Therefore, our stream must be able to give us Japanese characters and still must count octets. The HTTP stream will not, in general, give EOF condition at the end of data. That is not the only complication. Suppose the web server replied: Content-type: text/plain; charset=iso-2022-jp Content-transfer-encoding: chunked We should expect to read a sequence of chunks of the format: <length> CRLF <body> CRLF where <length> is a hexadecimal number, and <body> is encoded as indicated in the Content-type. Therefore, after we read the header, we should keep our stream in the ASCII mode to read the <length> field. After that, we should switch the encoding into ISO-2022-JP. After we have consumed <length> octets, we should switch the stream back to ASCII, verify the trailing CRLF, and read the <length> of the next chunk. The ISO-2022-JP encoding is stateful and wide (a character is formed by several octets). It may well happen that a wide character is split between two chunks: one octet of a character will be in one chunk and the other octets in following chunk. Therefore, when we switch from the ISO-2022-JP encoding to ASCII and back, we must preserve the state of the encoding. This is not the end of the story however. A web server may send us a multi-part reply: a multi-part MIME entity made of several MIME entities, each with its own encoding and transfer modes. Neither of these encoding have anything to do with the current locale. Therefore, we may need to switch encodings back and forth quite a few times. Decoding of such complex streams becomes easier if we can overlay different processing layers on a stream. We start with a TCP handle, overlay an ASCII stream and read the headers, then overlay a stream that reads a specified number of units (and returns EOF when read that many). On the top of the latter we place an ISO-2022-JP decoder. Or we choose a base64-decoder overlayed with a PCS7 signed entity decoder and with a signature verification layer. OpenSSL is one package that offers i/o overlays and stream composition. Overlaying of parsers, encoders and hash accumulators is very common in that particular domain. I have implemented such a facility in two languages, e.g., to overlay an endian stream on the top of a raw stream, then a bit stream and an arithmetic compression stream. In a functional world, Ensemble system [http://citeseer.nj.nec.com/liu99building.html] supports such a stackable i/o. When using overlayed streams, we should remember that all the layers are synchronized. If we did let iso2022_stream = makeiso2022 hFile in body then the raw hFile is still available in the body. Reading the stream with hFile and iso2022_stream indiscriminately will sure wreck havoc. Clean's unique types is an excellent feature to guard against such mistakes. Exactly the same considerations apply if we were using TIFFreader or PNG reader rather than ISO-2022-JP.

2 1

RE: [Haskell-cafe] Writing binary files?
by Simon Marlow 15 Sep '04

15 Sep '04

On 15 September 2004 12:32, ross(a)soi.city.ac.uk wrote: > On Mon, Sep 13, 2004 at 12:01:58PM +0100, Glynn Clements wrote: >> My view is that, right now, we have the worst of both worlds, and >> taking a short step backwards (i.e. narrow the Char type and leave >> the rest alone) is a lot simpler (and more feasible) than the long >> journey towards real I18N. > > This being Haskell, I can't imagine a consensus on a step backwards. > In any case, a Char type distinct from bytes and the rest is the most > valuable part of the current situation. The rest is just libraries, > and the solution to that is to create other libraries. (It's true > that the Prelude is harder to work around, but even that can be done, > as with the new exception interface.) Indeed more than one approach > can proceed concurrently, and that's probably what's going to happen: > > The Right Thing proceeds in stages: > 1. new byte-based libraries > 2. conversions sitting on top of these > 3. the ultimate I18N API > > The Quick Fix: alter the existing implementation to use the > encoding determined by the current locale at the borders. I wish I had some more time to work on this, but I implemented a prototype of an "ultimate" i18n API recently. This is derived from the API that Ben Rudiak-Gould proposed last year. It is in two layers: the InputStream/OutputStream classes provide raw byte I/O, and the TextStream class provides a conversion on top of that. The prototype uses iconv for conversions. You can make Streams from all sorts of things: files, sockets, pipes, and even Haskell arrays. InputStream and OutputStreams are just classes, so you can implement your own. IIRC, I managed to get it working with speed comparable to GHC's current IO library. Here's a tarball that works with GHC 6.2.1 on a Unix platform, just --make to build it: http://www.haskell.org/~simonmar/new-io.tar.gz If anyone would like to pick this up and run with it, I'd be delighted. I'm not likely to get back to in the short term, at least. Cheers, Simon

2 1

Re: [Haskell-cafe] Layered I/O
by MR K P SCHUPKE 15 Sep '04

15 Sep '04

My thoughts on I/O, binary and chars can be summerised: 1) Produce a new WordN based IO library. 2) Character strings cannot be separated from their encodings (ie they must be encoded somehow - even if that encoding is ascii). I would approch this using parameterised phantom types, so for example you would have: data Ascii data Chr a = Char Then the encoding becomes explicit in the type system, but of course functions like equality on types cvan still be expressed generically: charEq :: Chr a -> Chr a -> Bool but of course, you must be comparing the same encodings (enforced by type system)! The phantom type could be used in IO, IE to define a new encodeing: data MyEncoding instance ChrToBinary MyEncoding where chrToBinary = ... instance BinaryToChr MyEncoding where binaryToChr = ... Keean.

1 0

complete documentation like the one from Java or C++
by PRISACARIU Cristian 10 Sep '04

10 Sep '04

Hellow, can you tell me where I can find a more explicit documentation for the Haskell library and maybe for some of the extentions. GHC comes with something but is not as I want. I would like them to tell me for a function what it does not only what arguments it takes and what it returnes. And maybe some hints about how it works if not some examples. I would like something that approch the one from Java or .NET or MSDN. thank you for your time and understanding

5 4

(no subject)
by jcp＠cs.yale.edu 10 Sep '04

10 Sep '04

From: John Peterson <peterson-john(a)cs.yale.edu> To: haskell-cafe(a)haskell.org Subject: Re: [Haskell-cafe] Functional Reactive Programming Functional Reactive Programming is alive but in need of some new students to push the effort a bit. A lot of us have taken teaching or industrial positions so the old FRP team is a bit depleted. I don't think anyone is working on Yampa directly at the moment. Although it's stable and working well it lacks a critical mass of nice libraries to make it attractive. I'm still plugging on a wxHaskell port to Yampa (the wxFruit stuff). I've made some semantic changes to Yampa so I probably shouldn't say it's real Yampa but pretty close. I should have something to release later this fall. Aside from that, we have a student working in the hybrid modeling area. That's good stuff but not likely to produce software of interest to Joe Haskell. Another student is keeping the robotics side of things alive but it's in the context of a very specialized robotic hardware environment. So there you go! John

2 1

Re: [Haskell] Haskell recruitment?
by Lyle Kopnicky 09 Sep '04

09 Sep '04

I think there are so few such opportunities that it would not overwhelm the list. I should also point out that you can post such openings at http://www.haskell.org/jobs.html by sending an e-mail to peterson-john(a)cs.yale.edu and olaf(a)cs.york.ac.uk. Regards, Lyle Jochen L. Leidner wrote: >Finally, would it be considered spam to post Haskell-realted job postings >in this group? Thanks in advance. > > Regards, > Jochen > > >

2 1

cost of List.// for Ord types?
by David Roundy 08 Sep '04

08 Sep '04

Hello, I was wondering if the list diff operator \\ takes advantage of situations where the list data type is in class Ord, besides being in Eq. If it is only in Eq, then \\ must be O(n^2), while if it is in Ord, a O(nlogn) \\ can be written. Basically, I'm wondering if I should avoid using the standard library \\, and if so, whether there might be some better solution than writing my own (which would involve sorting the lists, and then the obvious picking off of the appropriate values). If there isn't a \\ that takes advantage of Ord, could there be? This being a more fundamental question: can you write a separate version of a function for the case where its argument is within a more specific class? -- David Roundy http://www.abridgegame.org

4 3

Functional Reactive Programming
by Mark Carroll 07 Sep '04

07 Sep '04

I was wondering, how much active development is done on FRP frameworks these days. What direction is it going in, and who are the users? I haven't seen much new on Yampa lately so I wondered how that was doing, or if it was thought largely finished and maybe something else was going on. For instance, maybe more of the activity is in developing wxFruit or suchlike? -- Mark

1 0

Installing HaXML,(using Hugs and WindowsXP)
by Tom Spencer 07 Sep '04

07 Sep '04

Has anyone successfully installed HaXML to be used with winhugs in XP. Using the build.bat included with HaXML it seems that it can only be installed for GHC. Any suggestions? Thanks Tom Spencer PS. Apologies that this is pretty vague

3 2