Simon Marlow wrote:
I've been working on adding proper Unicode support to Handle I/O in GHC, and I finally have something that's ready for testing. I've put a patchset here:
Yay! Comments below.
Comments/discussion please!
Do you expect Hugs will be able to pick up all of this?
The only change to the existing behaviour is that by default, text IO is done in the prevailing encoding of the system. Handles created by openBinaryFile use the Latin-1 encoding, as do Handles placed in binary mode using hSetBinaryMode.
Sounds very good and reasonable.
We provide a way to change the encoding for an existing Handle:
hSetEncoding :: Handle -> TextEncoding -> IO ()
and various encodings:
latin1, utf8, utf16, utf16le, utf16be, utf32, utf32le, utf32be, localeEncoding,
Will there also be something to handle the UTF-16 BOM marker? I'm not sure what the best API for that is, since it may or may not be present, but it should be considered -- and could perhaps help autodetect encoding.
Thanks to suggestions from Duncan Coutts, it's possible to call hSetEncoding even on buffered read Handles, and the right thing happens. So we can read from text streams that include multiple encodings, such as an HTTP response or email message, without having to turn buffering off (though there is a penalty for switching encodings on a buffered Handle, as the IO system has to do some re-decoding to figure out where it should start reading from again).
Sounds useful, but is this the bit that causes the 30% performance hit?
Performance is about 30% slower on "hGetContents >>= putStr" than before. I've profiled it, and about 25% of this is in doing the actual encoding/decoding, the rest is accounted for by the fact that we're shuffling around 32-bit chars rather than bytes in the Handle buffer, so there's not much we can do to improve this.
Does this mean that if we set the encoding to latin1, tat we should see performance 5% worse than present? 30% slower is a big deal, especially since we're not all that speedy now.
IO library restructuring ~~~~~~~~~~~~~~~~~~~~~~~~
The major change here is that the implementation of the Handle operations is separated from the underlying IO device, using type classes. File descriptors are just one IO provider; I have also implemented memory-mapped files (good for random-access read/write) and a Handle that pipes output to a Chan (useful for testing code that writes to a Handle). New kinds of Handle can be implemented outside the base package, for instance someone could write bytestringToHandle. A Handle is made using mkFileHandle:
Very nice. That means I can eliminate all the HVIO stuff I have in MissingH, which does roughly the same thing.
with making new kinds of Handle. We could split up the layers further later.
Would it now be possible to make the Socket an instance of this typeclass, so we can work with it directly rather than having to convert it to a Handle first? Thanks, -- John