On Mon, May 19, 2003 at 07:21:00PM +0100, Glynn Clements wrote:

George Russell wrote:

...

The general problem to solve is "How to we convert things to and from a binary format, so they can be efficiently written to and from disk".

Possibly the best way to handle this would be to extend the Storable class, e.g.

class Storable a where ... hPutObj :: Handle -> a -> IO () hGetObj :: Handle -> IO a toByteList :: a -> [Word8] fromByteList :: [Word8] -> a toByteArray :: a -> Array Int Word8 fromByteArray :: Array Int Word8 -> a

Hmmm, I thought that Foreign.Storable.Storable is meant for different purposes, namely for accessing C data in memory and for marshalling between Haskell and C worlds. Are you sure that these functionalities must overlap? Extending Storable would require to extend its every existing instance. It might be better to introduce some subclass, but I'm not sure that every type supporting marshalling to and from binary format can support operations from Storable. It's quite easy to write value of type [[Int]] to disk, but how to define alignment and sizeOf methods for this type? Best regards, Tom -- .signature: Too many levels of symbolic links

Glynn Clements writes:

George Russell wrote:

...

The general problem to solve is "How to we convert things to and from a binary format, so they can be efficiently written to and from disk". I have solved this (incompletely) twice myself,

Possibly the best way to handle this would be to extend the Storable class, e.g.

class Storable a where : [snip] : toByteArray :: a -> Array Int Word8 fromByteArray :: Array Int Word8 -> a

I wonder if this could be used to address a question (or perhaps "frustration" is a better term), namely that of efficient packing of data structures in memory. Would it be possible to separate binary I/O in two layers, the binary layer converting to/from (U?)Arrays of Word8, and another, the I/O layer, reading and writing such arrays? Something almost entirely, but not quite, unlike: class Binary a where toBin :: a -> Array Int Word8 toBinList :: [a] -> Array Int Word8 fromBin :: Array Int Word8 -> a -- perhaps with offset? fromBinList :: Array Int Word8 -> [a] hPutArray :: Handle -> Array Int Word8 -> IO () hGetArray :: ... writeBinaryFile :: FilePath -> Array Int Word8 -> IO () : -kzm -- If I haven't seen further, it is by standing in the footprints of giants

Ketil Z. Malde wrote (snipped)

Would it be possible to separate binary I/O in two layers, the binary layer converting to/from (U?)Arrays of Word8, and another, the I/O layer, reading and writing such arrays?

I think the proposal I posted is better because it allows you to mix up both single Word8's and arrays. In my experience, structures to be exported typically contain both. For example, I pack integers as a sequence of bytes (so that integers between -64 and 63 can be represented by one byte, for example). Constructing singleton arrays for each of these bytes would be tiresome, and presumably not as efficient as writing them directly into the target memory area. Here are two examples of cases where I want to write binary data to memory, not to a Handle: (1) Suppose I maintain a dictionary indexed by Strings (say) whose values belong to different types with binary representations, and I want this dictionary too to have a binary representation. Whenever I look something up in the dictionary I know what type I'm expecting, but I don't know what types to expect when decoding the dictionary. Then the most practical encoding seems to be as [(String,Bytes)], where each Bytes contains an encoded entry in the dictionary. (2) If I want to process binary data in some way before sending it to the outside world, for example by compressing it using zlib (the API to the GZIP algorithm), or by encrypting it.