Hello, Donald Bruce Stewart wrote:

Ok, I forgot one point. It is possible to automatically derive instances of Binary for your custom types, if they inhabit Data and Typeable, using an SYB trick. Load tools/derive/BinaryDerive.hs into ghci, and bring your type into scope, then run:

*Main> mapM_ putStrLn . lines $ derive (undefined :: Drinks)

It would seem that one needs to rerun the script every time the type is changed. That would be unfortunate. Perhaps I could have a go at writing a template haskell function to derive those instances? I also fear that the existing script does not handle types with more than 256 constructors correctly. While uncommon, those are not unrealistic. Using DrIFT would probably automate the deriving just as well, but in my particular situation TH support is easier to maintain than DrIFT support. Greetings, Arie

Joel Reymont wrote:

May I ask why TH is easier to maintain than DrIFT?

I'm not familiar with DrIFT.

The reason is personal, and very silly. I only use ghc, so TH is available automatically. Like you, I have never used DrIFT, so I would have to get to know it, and install it everywhere I want to compile my program. From a very quick look at the DrIFT homepage, installation might be nontrivial on a windows machine without some cygwin-like environment. At any rate, *for me* it's more work than using TH, because I'm familiar with the latter and already depend on its presence.

Why would I prefer one over the other?

I wouldn't know. Please do not let my prejudice influence your preference :-). Greetings, Arie

On Fri, Jan 26, 2007 at 03:12:29PM +0000, Dougal Stanton wrote:

Quoth Arie Peterson, nevermore,

...

I also fear that the existing script does not handle types with more than 256 constructors correctly. While uncommon, those are not unrealistic.

"256 constructors ought to be enough for anybody"? ;-)

Seriously though, the thought of a type definition that heavyweight quite terrifies me.

Think about simple enumerations, eg. for keywords in a programming language: data Keyword = IF | THEN | ELSE | BEGIN | END ... http://www.cs.vu.nl/grammars/cobol/: Number of keywords: 420 Perhaps such examples could be treated differently, but I think it's better to have a more general solution and not have to assume unneccesary restrictions on user's datatypes.

I would be interested to see if such a thing could be warranted and not more sensibly broken down into smaller (sets of) units.

I think in the above example the most sensible thing is to have all the keywords in the same datatype. Best regards Tomasz

On Fri, 2007-01-26 at 15:40 +0100, Arie Peterson wrote:

Hello,

Donald Bruce Stewart wrote:

...

Ok, I forgot one point. It is possible to automatically derive instances of Binary for your custom types, if they inhabit Data and Typeable, using an SYB trick. Load tools/derive/BinaryDerive.hs into ghci, and bring your type into scope, then run:

*Main> mapM_ putStrLn . lines $ derive (undefined :: Drinks)

It would seem that one needs to rerun the script every time the type is changed. That would be unfortunate. Perhaps I could have a go at writing a template haskell function to derive those instances?

I also fear that the existing script does not handle types with more than 256 constructors correctly. While uncommon, those are not unrealistic.

Feel free to send in a patch. All it needs to do is check if there are more than 2^8 constructors and if so encode the tag in a Word16 rather than Word8. Duncan

On Fri, Jan 26, 2007 at 02:16:22PM +1100, Donald Bruce Stewart wrote:

We believe so, and its a bug if this is not the case.

The src documents the encoding format used for each type (we were unable to attach haddocks to instances.. grr.)

All data is encoded in Network order, and extended to 64 bits for word sized values (like Int). It should be possible to encode a structure with ghc on x86, and decode it on a sparc64 running hugs.

Did you consider using an encoding which uses variable number of bytes? If yes, I would be interested to know your reason for not choosing such an encoding. Efficiency? Best regards Tomasz

existing ecoding system - both the BER (Basic Encoding Rules) and the PER (Packed Encoding Rules). If you are looking to target a well supported standard - this would be the one. Neil On 26/01/07, Malcolm Wallace wrote:

Tomasz Zielonka wrote:

...

Did you consider using an encoding which uses variable number of bytes? If yes, I would be interested to know your reason for not choosing such an encoding. Efficiency?

My Binary implementation (from 1998) used a type-specific number of bits to encode the constructor - exactly as many as needed. (If you were writing custom instances, you could even use a variable number of bits for the constructor, e.g. using Huffman encoding to make the more common constructors have the shortest representation.)

The latter certainly imposes an extra time overhead on decoding, because you cannot just take a fixed-size chunk of bits and have the value. But I would have thought that in the regular case, using a type-specific (but not constructor-specific) size for representing the constructor would be very easy and have no time overhead at all.

Regards, Malcolm _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On Fri, Jan 26, 2007 at 04:36:50PM +0100, Tomasz Zielonka wrote:

Did you consider using an encoding which uses variable number of bytes? If yes, I would be interested to know your reason for not choosing such an encoding. Efficiency?

I am testing/benchmarking one right now I wrote for 'Integer', so far, I think it may be better in time _and_ space! cache effects no doubt. A nice thing about it is that for the common case, short ascii strings, the serialized form takes up exactly as much as they would in C, very nice. :) John -- John Meacham - ⑆repetae.net⑆john⑈

On Thu, Jan 25, 2007 at 07:11:55PM -0800, John Meacham wrote:

Is the binary format portable? I need the produced files to work on both 32 and 64 bit architectures and with big and little endian machines. And of course, between different versions of a compiler or different compilers.

Sorry to reply to myself, looking at the code, I see that it is. however, Ints appear to be stored as 64 bits always, this seems like a mistake. The Haskell standard only specifies Ints must have at least 30 bits of precision so programs that rely on more than that are not portable anyway. Plus, it is unlikely that any compilers ever will have Ints > 32 bits, ghc does at the moment by accident of design and it is considered a misfeature that will be fixed at some point. It would be an ugly wart to be stuck with going forward... John -- John Meacham - ⑆repetae.net⑆john⑈

On Fri, Jan 26, 2007 at 04:42:48PM +0100, Tomasz Zielonka wrote:

I also have to use a specific serialisation format. I guess we could both simply use putWord8, but then we'll probably lose most of the benefits of using the library.

Perhaps we could think about introducing some "encoding contexts", with a default encoding that can be automatically derived, but also with the ability to create one's own encodings?

one can use newtypes they would be faster in any case, I was thinking something like:

newtype XDRInt = XDRInt Int newtype XDRSTring sz = ...

and so forth, now if you build up a structure

data NfsSattr = NfsSattr { mode :: XdrUnsigend, -- protection mode bits uid :: XdrUnsigned, -- owner user id gid :: XdrUnsigned, -- owner group id size :: XdrUnsigned, -- file size in bytes atime :: XdrNfsTime, -- time of last access mtime :: XdrNfsTime -- time of last modification }

now you can speak nfs directly by serializing right from and to your socket! :) a whole filesystem implemented in haskell in not so many lines. very nice. actually, I probably will write Data.Binary.Protocol.Xdr (better location?). I actually do have a NFS server written in haskell in a much more clunky way I could revive. now, the only new primitives I would need are:

alignTo :: Word8 -> Int -> Put alignTo _ _ = ...

setAlignment :: Int -> Put setAlignment _ = ...

where alignTo would output some number of bytes in order to bring the stream to the next alignment boundry specified, and setAlignment would force the current alignment to be some value, without outputing any bytes. Would these be doable? They would open up a lot of possibilities. John -- John Meacham - ⑆repetae.net⑆john⑈

tomasz.zielonka:

On Fri, Jan 26, 2007 at 04:31:28PM +0100, Henning Thielemann wrote:

...

On Fri, 26 Jan 2007, Donald Bruce Stewart wrote:

...

Binary: high performance, pure binary serialisation for Haskell ----------------------------------------------------------------------

The Binary Strike Team is pleased to announce the release of a new, pure, efficient binary serialisation library for Haskell, now available from Hackage:

tarball: http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary/0.2 darcs: darcs get http://darcs.haskell.org/binary haddocks: http://www.cse.unsw.edu.au/~dons/binary/Data-Binary.html

I want to write out data in the machine's endianess, because that data will be post-processed by sox, which reads data in the machine's endianess. Is this also planned for the package?

I also have to use a specific serialisation format. I guess we could both simply use putWord8, but then we'll probably lose most of the benefits of using the library.

Perhaps we could think about introducing some "encoding contexts", with a default encoding that can be automatically derived, but also with the ability to create one's own encodings?

Note that using Binary directly for non-Haskell structures is a bit like using Read/Show instances for parsing non-Haskell structures: possible, but not optimal. It would be better to use the underlying Get/Put monads available in binary, with the low level support for explicit endian and word sized writes/reads, to build a combinator library on top for these more flexible parsing/binary requirements (like layering Parsec over ReadP or HughesPJ). -- Don

Donald Bruce Stewart wrote:

lemming:

...

On Fri, 26 Jan 2007, Donald Bruce Stewart wrote:

...

Binary: high performance, pure binary serialisation for Haskell ----------------------------------------------------------------------

The Binary Strike Team is pleased to announce the release of a new, pure, efficient binary serialisation library for Haskell, now available from Hackage:

tarball: http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary/0.2 darcs: darcs get http://darcs.haskell.org/binary haddocks: http://www.cse.unsw.edu.au/~dons/binary/Data-Binary.html

I want to write out data in the machine's endianess, because that data will be post-processed by sox, which reads data in the machine's endianess. Is this also planned for the package?

The underlying Get and Put monads support explicit endian writes and reads, which you can add to your instances explicitly:

http://www.cse.unsw.edu.au/~dons/binary/Data-Binary-Get.html#5 http://www.cse.unsw.edu.au/~dons/binary/Data-Binary-Put.html#5

So you can do that now.

-- Don

The documentation has a small organization bug: http://www.cse.unsw.edu.au/~dons/binary/Data-Binary-Get.html#5 The "Big-endian reads" section has both big endian and little endian functions. Same for the "Little-endian reads" section. The page for Put is okay. -- Chris

On Sat, 2007-01-27 at 19:11 +0100, Henning Thielemann wrote:

On Sat, 27 Jan 2007, Donald Bruce Stewart wrote:

...

The underlying Get and Put monads support explicit endian writes and reads, which you can add to your instances explicitly:

http://www.cse.unsw.edu.au/~dons/binary/Data-Binary-Get.html#5 http://www.cse.unsw.edu.au/~dons/binary/Data-Binary-Put.html#5

So you can do that now.

Of course, I can check for the machine's endianess, then decide whether to use putWord16be and putWord16le. But I assume that it is more efficient, if there would be some function putWord16native, which writes out in the machine's native endianess.

Note that actually taking advantage of aligned word reads/writes is not especially simple since one has to guarantee alignment. So that means no byte writes, everything, chars, bools, record tags would have to be word sized. While it's not so hard to arrange alignment for writes (since we get to allocate the buffers) for reads we have no guarantee since the user supplies the byte string input and they could have uses say 'tail' so the data may be all unaligned. Dealing with both aligned and unaligned makes it more complex. We did a performance experiment with this and found that it gained us very little. There are bigger performance bottlenecks at the moment than the penalty of always doing byte reads/writes. Duncan

dons:

lemming:

...

On Sat, 27 Jan 2007, Donald Bruce Stewart wrote:

...

lemming:

...

On Fri, 26 Jan 2007, Donald Bruce Stewart wrote:

...

Binary: high performance, pure binary serialisation for Haskell ----------------------------------------------------------------------

The Binary Strike Team is pleased to announce the release of a new, pure, efficient binary serialisation library for Haskell, now available from Hackage:

tarball: http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary/0.2 darcs: darcs get http://darcs.haskell.org/binary haddocks: http://www.cse.unsw.edu.au/~dons/binary/Data-Binary.html

I want to write out data in the machine's endianess, because that data will be post-processed by sox, which reads data in the machine's endianess. Is this also planned for the package?

The underlying Get and Put monads support explicit endian writes and reads, which you can add to your instances explicitly:

http://www.cse.unsw.edu.au/~dons/binary/Data-Binary-Get.html#5 http://www.cse.unsw.edu.au/~dons/binary/Data-Binary-Put.html#5

So you can do that now.

Of course, I can check for the machine's endianess, then decide whether to use putWord16be and putWord16le. But I assume that it is more efficient, if there would be some function putWord16native, which writes out in the machine's native endianess.

I've added raw primitives for: {put,get}Wordhost {put,get}Word16host {put,get}Word32host {put,get}Word64host which do unaligned, host-sized, host-endian packing of data. Writing is some 15% faster for Words, a bit less for reading (which is a bit unoptimised at the moment), and a bit less for other types. Needless to say, data serialised this way is only portable to the same architecture. Now available from specially selected bookshops and darcs repositories. Please enjoy responsibly! -- Don

Hi Conrad,

If the data header stores the alignment/size/endianness, then there's no reason for the data to be unportable. The normal get* instances (not get*host) could suffice for reading.

That requires the stream to have a header. Which means that any arbitrary slice within the ByteString is not equal - the first one contains essential information which isn't available anywhere else. It seems like a lot of complexity, defining multiple parameterised file types, when the simple case is probably neater and cheaper in terms of both runtime and developer time. Thanks Neil

On Jan 29, 2007, at 16:38 , Ross Paterson wrote:

On Fri, Jan 26, 2007 at 01:51:01PM +1100, Donald Bruce Stewart wrote:

...

Binary: high performance, pure binary serialisation for Haskell

--------------------------------------------------------------------- -

The Binary Strike Team is pleased to announce the release of a new, pure, efficient binary serialisation library for Haskell, now available from Hackage:

tarball: http://hackage.haskell.org/cgi-bin/hackage-scripts/ package/binary/0.2 darcs: darcs get http://darcs.haskell.org/binary haddocks: http://www.cse.unsw.edu.au/~dons/binary/Data-Binary.html

Remind me again: why do you need a Put monad, which always seems to have the argument type ()? Monoids really are underappreciated.

I don't know if that's the main reason, but monads give you do- notation, which can be very nice when you are doing a long sequence of puts. Here's an example from the tar package [1]: putHeaderNoChkSum :: TarHeader -> Put putHeaderNoChkSum hdr = do let (filePrefix, fileSuffix) = splitLongPath 100 (tarFileName hdr) putString 100 $ fileSuffix putOct 8 $ tarFileMode hdr putOct 8 $ tarOwnerID hdr putOct 8 $ tarGroupID hdr putOct 12 $ tarFileSize hdr putOct 12 $ let TOD s _ = tarModTime hdr in s fill 8 $ ' ' -- dummy checksum putTarFileType $ tarFileType hdr putString 100 $ tarLinkTarget hdr -- FIXME: take suffix split at / if too long putString 6 $ "ustar " putString 2 $ " " -- strange ustar version putString 32 $ tarOwnerName hdr putString 32 $ tarGroupName hdr putOct 8 $ tarDeviceMajor hdr putOct 8 $ tarDeviceMinor hdr putString 155 $ filePrefix fill 12 $ '\NUL' I guess mconcat [putX, putY, ... ] would work too, but the syntax is not quite as nice. /Björn [1] http://www.cs.chalmers.se/~bringert/darcs/tar/

On Tue, Jan 30, 2007 at 09:52:01AM +1100, Donald Bruce Stewart wrote:

For the syntax, and So that people can directly port their code from NewBinary. (The instances are basically unchanged).

newtype PutM a = Put { unPut :: (a, Builder) } type Put = PutM ()

It is always ().

BTW, the PutM type constructor is not exported, but I think it may be useful in some uses on the library. Best regards Tomasz

On Tue, Jan 30, 2007 at 09:52:01AM +1100, Donald Bruce Stewart wrote:

ross:

...

why do you need a Put monad, which always seems to have the argument type ()? Monoids really are underappreciated.

For the syntax, and So that people can directly port their code from NewBinary. (The instances are basically unchanged).

And so the successor to binary must have the same interface, and so on forever. The backward compatibility argument seems weak to me, leaving only the advantage of do-notation. Monads are the new lists.

On Tue, Jan 30, 2007 at 10:22:58AM +0000, Duncan Coutts wrote:

I was about to say that for the more complicated binary serialisation formats (eg GHC's .hi format) people need monads with state, like string pools etc, but actually now that I think about it, that can be done with a monoid too.

Ross, you need to make a monoid transformer library (at least reader and state) and campaign for ++ to be redefined as mappend, then everyone will want to use it since it'll be so neat and convenient! :-)

Can anyone see a real serialisation use case that needs a monad for the serialisation side? I'd thought I had an example, but I was wrong.

If you really want a monad, you can put the monoid into the writer monad - and that's exactly what is done in Data.Binary! However, for someone who prefers the monadic interface, wrapping in the writer monad every time and additional boilerplate code (tell) can be a nuisance. Best regards Tomasz

Hello Duncan, Tuesday, January 30, 2007, 1:22:58 PM, you wrote:

Can anyone see a real serialisation use case that needs a monad for the serialisation side? I'd thought I had an example, but I was wrong.

my program, FreeArc, has its own compression level on top of serializing - i.e. data serialized sent in 64k blocks to the C compression routine and both serialization and compression are run at the same time using threads -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Hello Duncan, Thursday, February 1, 2007, 3:39:16 AM, you wrote:

...

...

Can anyone see a real serialisation use case that needs a monad for the serialisation side? I'd thought I had an example, but I was wrong.

my program, FreeArc, has its own compression level on top of serializing - i.e. data serialized sent in 64k blocks to the C compression routine and both serialization and compression are run at the same time using threads

i mean that in real world, programs may need to do something in IO monad - work with database, network, call C libs -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

On Tue, Jan 30, 2007 at 09:38:26AM +0000, Ross Paterson wrote:

On Tue, Jan 30, 2007 at 09:52:01AM +1100, Donald Bruce Stewart wrote:

...

ross:

...

why do you need a Put monad, which always seems to have the argument type ()? Monoids really are underappreciated.

For the syntax, and So that people can directly port their code from NewBinary. (The instances are basically unchanged).

And so the successor to binary must have the same interface, and so on forever. The backward compatibility argument seems weak to me, leaving only the advantage of do-notation.

How about having both interfaces, so you can use the one you like better? class Binary t where -- | Encode a value in the Put monad. put :: t -> Put put x = Put ((), build x) -- | Encode a value using the Builder monoid build :: t -> Builder build x = snd (unPut (put x)) -- | Decode a value in the Get monad get :: Get t The downside is that GHC probably wouldn't warn about undefined methods, or would it? Best regards Tomasz

simonmar:

Donald Bruce Stewart wrote:

...

Binary: high performance, pure binary serialisation for Haskell ---------------------------------------------------------------------- The Binary Strike Team is pleased to announce the release of a new, pure, efficient binary serialisation library for Haskell, now available from Hackage:

tarball: http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary/0.2 darcs: darcs get http://darcs.haskell.org/binary haddocks: http://www.cse.unsw.edu.au/~dons/binary/Data-Binary.html

A little benchmark I had lying around shows that this Binary library beats the one in GHC by a factor of 2 (at least on this example):

Very nice. We've been benchmarking again NewBinary, for various Word-sized operations, with the following results, on x86: NewBinary, fairly tuned (lots of fastMutInt#s) 10MB of Word8 in chunks of 1: 10.68MB/s write, 9.16MB/s read 10MB of Word16 in chunks of 16: 7.89MB/s write, 6.65MB/s read 10MB of Word32 in chunks of 16: 7.99MB/s write, 7.29MB/s read 10MB of Word64 in chunks of 16: 5.10MB/s write, 5.75MB/s read Data.Binary: 10MB of Word8 in chunks of 1 ( Host endian): 11.7 MB/s write, 2.4 MB/s read 10MB of Word16 in chunks of 16 ( Host endian): 89.3 MB/s write, 3.6 MB/s read 10MB of Word16 in chunks of 16 ( Big endian): 83.3 MB/s write, 1.6 MB/s read 10MB of Word32 in chunks of 16 ( Host endian): 178.6 MB/s write, 7.2 MB/s read 10MB of Word32 in chunks of 16 ( Big endian): 156.2 MB/s write, 2.5 MB/s read 10MB of Word64 in chunks of 16 ( Host endian): 78.1 MB/s write, 11.3 MB/s read 10MB of Word64 in chunks of 16 ( Big endian): 44.6 MB/s write, 2.8 MB/s read Note that we're much faster writing, in general, but read speed lags. The 'get' monad hasn't received much attention yet, though we know what needs tuning.

GHC's binary library (quite heavily tuned by me):

Write time: 2.41 Read time: 1.44 1,312,100,072 bytes allocated in the heap 96,792 bytes copied during GC (scavenged) 744,752 bytes copied during GC (not scavenged) 32,492,592 bytes maximum residency (6 sample(s))

2384 collections in generation 0 ( 0.01s) 6 collections in generation 1 ( 0.00s)

63 Mb total memory in use

INIT time 0.00s ( 0.00s elapsed) MUT time 3.78s ( 3.84s elapsed) GC time 0.02s ( 0.02s elapsed) EXIT time 0.00s ( 0.00s elapsed) Total time 3.79s ( 3.86s elapsed)

Data.Binary:

Write time: 0.99 Read time: 0.65 1,949,205,456 bytes allocated in the heap 204,986,944 bytes copied during GC (scavenged) 5,154,600 bytes copied during GC (not scavenged) 70,247,720 bytes maximum residency (8 sample(s))

3676 collections in generation 0 ( 0.25s) 8 collections in generation 1 ( 0.19s)

115 Mb total memory in use

INIT time 0.00s ( 0.00s elapsed) MUT time 1.08s ( 1.13s elapsed) GC time 0.44s ( 0.52s elapsed) EXIT time 0.00s ( 0.00s elapsed) Total time 1.51s ( 1.65s elapsed)

This example writes a lot of 'Maybe Int' values. I'm surprised by the extra heap used by Data.Binary: this was on a 64-bit machine, so Ints should have been encoded as 64 bits by both libraries. Also, the GC seems to be working quite hard with Data.Binary, I'd be interested to know why that is.

Very interesting! Is this benchmark online? I'm a little surprised by the read times, reading is still fairly unoptimised compared to writing.

Anyway, this result is good enough for me, I'd like to use Data.Binary in GHC as soon as we can. Unfortunately we have to support older compilers, so there will be some build-system issues to surmount. Also we need a way to pass state around while serialising/deserialising - what's the current plan for this?

The plan was to use StateT Put or StateT Get, I think. But we don't have a demo for this yet. Duncan, Lennart, any suggestions? -- Don