Reading/Writing Binary Data in Haskell
Hello to all, I'm recently working on doing some atmospheric modelling for my PhD thesis work and I've been writing parallel implementations in Java, Ruby, and Haskell. I picked up Haskell as part of the LoTY project and was especially impressed by how expressive and clean the code is. I've been quite successful at writing most of my numerical libraries in Haskell (typically at a much reduced SLOC) but I'm not butting my head against binary data. I have a large store of data, output by a Fortran model, which essentially consists of binary files containing arrays of floating point values. In doing some googling on reading binary data in Haskell I've come across some old (> 1 year) references to some mailing list discussions on reading binary data. So given the background I have a few questions: 1 - Is there yet a standard, or at least commonly supported by hugs and ghc, method for dealing with binary data. 2 - If not, is there a "standard" library that is used to manipulate binary data. I've seen some references to some implementations but given that this is being done in my limited spare time (I'm not a full time student) I'd rather not spin my wheels on something that's going to be dead in a matter of months. 3 - What is the current status of getting binary data into the Haskell standard? I understand that I can write a C extension, but part of my interest in Haskell lies in the ability to write scientific software that very closely resembles the mathematics that are being modelled. On that note, I'd also be very interested in conversing with people that are doing physics modelling (atmospheric dynamics and chemistry is esp. interesting) using functional languages. Thanks! Gordon Miller gmiller@bittwiddlers.com
Gordon James Miller wrote:
1 - Is there yet a standard, or at least commonly supported by hugs and ghc, method for dealing with binary data.
2 - If not, is there a "standard" library that is used to manipulate binary data. I've seen some references to some implementations but given that this is being done in my limited spare time (I'm not a full time student) I'd rather not spin my wheels on something that's going to be dead in a matter of months.
There isn't a standard mechanism for binary I/O. However, a simple and fairly generic mechanism for doing this is: 1. Read in a list of "Char"s with the standard I/O functions. 2. Use "map (fromIntegral . ord) ..." to get a list of "Word8"s (octets). 3. Use with, poke, peek, castPtr etc to coerce a list of octets to the desired type(s). For output, do the reverse. This way, you can read/write any instance of Storable in the host's native format (i.e. "C" format), and the code should be portable. The key here is step 3; if you want to sacrifice portability for performance, use Posix.read (or the Win32 equivalent) to read directly into memory (Ptr/Addr), or even write an import declaration for mmap(). -- Glynn Clements <glynn.clements@virgin.net>
On Wednesday, 2003-07-09, 05:31, Glynn Clements wrote:
[...]
There isn't a standard mechanism for binary I/O.
NHC98 contains the York Binary library. Can someone tell me if this is available for other Haskell systems? And didn't GHC also provide binary I/O?
However, a simple and fairly generic mechanism for doing this is:
1. Read in a list of "Char"s with the standard I/O functions.
This will most likely cause problems under Windows. The reason is that the standard I/O functions are intended for reading and writing text, and that's why under Windows the sequence CRLF (#0D#0A) is interpreted as a single LF (#0A) and EOF (#1A) is interpreted as the end of the file.
2. Use "map (fromIntegral . ord) ..." to get a list of "Word8"s (octets).
This assumes that the standard I/O functions use a one byte character encoding. I don't now whether this is guaranteed by the Haskell report.
[...]
Wolfgang
There isn't a standard mechanism for binary I/O.
NHC98 contains the York Binary library. Can someone tell me if this is available for other Haskell systems? And didn't GHC also provide binary I/O?
How does the GHC itself read/write binary data, since the interface files (*.hi) produced by GHC are in a binary format? (Whereas NHC98 *.hi are text files.) Christian
And didn't GHC also provide binary I/O?
http://www.haskell.org/ghc/docs/latest/html/base/Data.Array.IO.html#4 -- -- Johannes Waldmann ---- http://www.informatik.uni-leipzig.de/~joe/ -- -- joe@informatik.uni-leipzig.de -- phone/fax (+49) 341 9732 204/209 --
Wolfgang Jeltsch wrote:
However, a simple and fairly generic mechanism for doing this is:
1. Read in a list of "Char"s with the standard I/O functions.
This will most likely cause problems under Windows. The reason is that the standard I/O functions are intended for reading and writing text, and that's why under Windows the sequence CRLF (#0D#0A) is interpreted as a single LF (#0A) and EOF (#1A) is interpreted as the end of the file.
Good point. Both GHC and Hugs provide openFileEx, which allows files to be read in binary mode (without EOL/EOF translations). -- Glynn Clements <glynn.clements@virgin.net>
On Wednesday, 2003-07-09, 15:16, CEST, Glynn Clements wrote:
[...]
Both GHC and Hugs provide openFileEx, which allows files to be read in binary mode (without EOL/EOF translations).
So we have portable binary I/O, don't we? By the way, does one still read characters rather than bytes even if using openFileEx? Is it guaranteed that one character corresponds to one byte when using openFileEx? Wolfgang
Wolfgang Jeltsch wrote:
Both GHC and Hugs provide openFileEx, which allows files to be read in binary mode (without EOL/EOF translations).
So we have portable binary I/O, don't we?
By the way, does one still read characters rather than bytes even if using openFileEx?
openFileEx returns a Handle just like openFile, and functions such as hGetChar return a Char regardless of where the Handle came from. OTOH, those "Char"s are still just octets stored in an oversized container.
Is it guaranteed that one character corresponds to one byte when using openFileEx?
The report says almost nothing about the semantics of I/O. AFAICT, an implementation could write the data in EBCDIC without violating the letter of the report. OTOH, existing implementations (at least GHC and Hugs) currently read and write "8-bit binary", i.e. characters 0-255 get read and written "as-is" and anything else breaks, and changing that would probably break a fair amount of existing code. -- Glynn Clements <glynn.clements@virgin.net>
tor 2003-07-10 klockan 04.56 skrev Glynn Clements:
OTOH, existing implementations (at least GHC and Hugs) currently read and write "8-bit binary", i.e. characters 0-255 get read and written "as-is" and anything else breaks, and changing that would probably break a fair amount of existing code.
What I would like to see, is a package for converting between different encodings and character sets. Python has two types for strings, 'str' (which is just a sequence of octets) and 'unicode'. You can encode and decode between them, I find this pretty neat: 'foo åäö'.decode('latin1') -> unicode string ustr.encode('latin1') -> string, breaks if there are non-latin1 characters in the string ustr.encode('utf-8') -> UTF-8 representation of the string. If I recall correctly, the 'str' type is being replaced with another type to highlight that it's actually only a sequence of bytes, whereas 'unicode' are Really Nice strings... Having something like this in Haskell would be wonderful, unfortunately I don't know much about Unicode beyond happily using it, so I don't have any suggestions or anything. :) /Martin -- Martin Sjögren martin@strakt.com Phone: +46 (0)31 7490880 Cell: +46 (0)739 169191 GPG key: http://www.strakt.com/~martin/gpg.html
On Thu, Jul 10, 2003 at 09:28:38AM +0200, Martin Sjgren wrote:
tor 2003-07-10 klockan 04.56 skrev Glynn Clements:
OTOH, existing implementations (at least GHC and Hugs) currently read and write "8-bit binary", i.e. characters 0-255 get read and written "as-is" and anything else breaks, and changing that would probably break a fair amount of existing code.
What I would like to see, is a package for converting between different encodings and character sets. Python has two types for strings, 'str' (which is just a sequence of octets) and 'unicode'.
Char in Haskell can represent Unicode and bytes are Word8. What is a bad legacy is that functions like withCString deal with bytes not with Unicode (or rather UTF8). I use the following as a workaround until there is some result from the internationalization effort for Haskell (http://sourceforge.net/projects/haskell-i18n/). Cheers, Axel. {-# OPTIONS -cpp #-} -- * This module adds CString-like functions that handle UTF8 strings. -- Furthermore it serves as an impedance matcher for different compiler -- versions. module FFI( with, nullForeignPtr, foreignFree, withUTFString, withUTFStringLen, newUTFString, newUTFStringLen, peekUTFString, peekUTFStringLen, module Foreign, #if __GLASGOW_HASKELL__>=504 module Foreign.C #else module CForeign #endif ) where import Monad (liftM) import Char import LocalData(unsafePerformIO) #if __GLASGOW_HASKELL__>=504 import Data.Bits import Foreign.C import qualified Foreign import Foreign hiding (with) #else import Bits import CForeign import qualified Foreign import Foreign hiding (withObject) #endif #if __GLASGOW_HASKELL__>=504 with :: (Storable a) => a -> (Ptr a -> IO b) -> IO b with = Foreign.with #else with :: (Storable a) => a -> (Ptr a -> IO b) -> IO b with = Foreign.withObject #endif #if __GLASGOW_HASKELL__>=600 foreign import ccall unsafe "&free" free' :: FinalizerPtr a foreignFree :: Ptr a -> FinalizerPtr a foreignFree _ = free' nullForeignPtr :: ForeignPtr a nullForeignPtr = unsafePerformIO $ newForeignPtr nullPtr free' #else nullForeignPtr :: ForeignPtr a nullForeignPtr = unsafePerformIO $ newForeignPtr nullPtr (return ()) foreignFree :: Ptr a -> IO () foreignFree = free #endif -- Define withUTFString to emit UTF-8. -- withUTFString :: String -> (CString -> IO a) -> IO a withUTFString hsStr = withCString (toUTF hsStr) -- Define withUTFStringLen to emit UTF-8. -- withUTFStringLen :: String -> (CStringLen -> IO a) -> IO a withUTFStringLen hsStr = withCStringLen (toUTF hsStr) -- Define newUTFString to emit UTF-8. -- newUTFString :: String -> IO CString newUTFString = newCString . toUTF -- Define newUTFStringLen to emit UTF-8. -- newUTFStringLen :: String -> IO CStringLen newUTFStringLen = newCStringLen . toUTF -- Define peekUTFString to retrieve UTF-8. -- peekUTFString :: CString -> IO String peekUTFString strPtr = liftM fromUTF $ peekCString strPtr -- Define peekUTFStringLen to retrieve UTF-8. -- peekUTFStringLen :: CStringLen -> IO String peekUTFStringLen strPtr = liftM fromUTF $ peekCStringLen strPtr -- Convert Unicode characters to UTF-8. -- toUTF :: String -> String toUTF [] = [] toUTF (x:xs) | ord x<=0x007F = x:toUTF xs | ord x<=0x07FF = chr (0xC0 .|. ((ord x `shift` (-6)) .&. 0x1F)): chr (0x80 .|. (ord x .&. 0x3F)): toUTF xs | otherwise = chr (0xE0 .|. ((ord x `shift` (-12)) .&. 0x0F)): chr (0x80 .|. ((ord x `shift` (-6)) .&. 0x3F)): chr (0x80 .|. (ord x .&. 0x3F)): toUTF xs -- Convert UTF-8 to Unicode. -- fromUTF :: String -> String fromUTF [] = [] fromUTF (all@(x:xs)) | ord x<=0x7F = x:fromUTF xs | ord x<=0xBF = err | ord x<=0xDF = twoBytes all | ord x<=0xEF = threeBytes all | otherwise = err where twoBytes (x1:x2:xs) = chr (((ord x1 .&. 0x1F) `shift` 6) .|. (ord x2 .&. 0x3F)):fromUTF xs twoBytes _ = error "fromUTF: illegal two byte sequence" threeBytes (x1:x2:x3:xs) = chr (((ord x1 .&. 0x0F) `shift` 12) .|. ((ord x2 .&. 0x3F) `shift` 6) .|. (ord x3 .&. 0x3F)):fromUTF xs threeBytes _ = error "fromUTF: illegal three byte sequence" err = error "fromUTF: illegal UTF-8 character"
participants (7)
-
Axel Simon -
Christian Maeder -
Glynn Clements -
Gordon James Miller -
Johannes Waldmann -
Martin Sjögren -
Wolfgang Jeltsch