Alternative to Data.Binary

Hi all, Is there a serialization library other than the Data.Binary from hackage? I am using Data.Binary in a couple of projects, but I have found its stack and memory usage very hard to control. Its very common that decoding a map or list of non-trivial size uses up all available RAM, or causes a stack overflow. I don't care that much about it being extremely fast, I just want to stop worrying that if I try to read a file a few percent larger than the last time, my program will suddenly stop working. Best, -- Grzegorz -- View this message in context: http://www.nabble.com/Alternative-to-Data.Binary-tp22512229p22512229.html Sent from the Haskell - Haskell-Cafe mailing list archive at Nabble.com.

Grzegorz Chrupala wrote:
Hi all, Is there a serialization library other than the Data.Binary from hackage?
Yes. "binary-strict" is one alternative: http://hackage.haskell.org/cgi-bin/hackage-scripts/package/binary-strict
I am using Data.Binary in a couple of projects, but I have found its stack and memory usage very hard to control. Its very common that decoding a map or list of non-trivial size uses up all available RAM, or causes a stack overflow.
I don't care that much about it being extremely fast, I just want to stop worrying that if I try to read a file a few percent larger than the last time, my program will suddenly stop working.
Best, -- Grzegorz

On Sat, Mar 14, 2009 at 1:37 PM, Grzegorz Chrupala
Hi all, Is there a serialization library other than the Data.Binary from hackage?
I am using Data.Binary in a couple of projects, but I have found its stack and memory usage very hard to control. Its very common that decoding a map or list of non-trivial size uses up all available RAM, or causes a stack overflow.
That little problem appears to be an artifact of the particular Binary instance for lists (the map instance starts by converting to/from a list), and should be fixable in principle, for example by writing them in chunks of 256 elements. That said, the current instance really should only have problems when *writing*, in that it'll force the entire list to figure out its length before it starts writing any of it. That this is not, in fact, the case suggests that the reader should be fixable without changing its on-disk format. -- Svein Ove Aas

Svein Ove Aas ha scritto:
On Sat, Mar 14, 2009 at 1:37 PM, Grzegorz Chrupala
wrote: Hi all, Is there a serialization library other than the Data.Binary from hackage?
I am using Data.Binary in a couple of projects, but I have found its stack and memory usage very hard to control. Its very common that decoding a map or list of non-trivial size uses up all available RAM, or causes a stack overflow.
That little problem appears to be an artifact of the particular Binary instance for lists (the map instance starts by converting to/from a list), and should be fixable in principle, for example by writing them in chunks of 256 elements.
I can confirm that reading serialized UArr from uvector package does not cause memory problems. Manlio

grzegorz.chrupala:
Hi all, Is there a serialization library other than the Data.Binary from hackage?
I am using Data.Binary in a couple of projects, but I have found its stack and memory usage very hard to control. Its very common that decoding a map or list of non-trivial size uses up all available RAM, or causes a stack overflow.
I don't care that much about it being extremely fast, I just want to stop worrying that if I try to read a file a few percent larger than the last time, my program will suddenly stop working.
Have you tried the latest release, which modified the Map and [a] instances? -- Don

Don Stewart-2 wrote:
Have you tried the latest release, which modified the Map and [a] instances?
No, I'm working with 0.5. I'll give the new version a try. Thanks! -- Grzegorz -- View this message in context: http://www.nabble.com/Alternative-to-Data.Binary-tp22512229p22514771.html Sent from the Haskell - Haskell-Cafe mailing list archive at Nabble.com.

Don Stewart ha scritto:
grzegorz.chrupala:
Hi all, Is there a serialization library other than the Data.Binary from hackage?
I am using Data.Binary in a couple of projects, but I have found its stack and memory usage very hard to control. Its very common that decoding a map or list of non-trivial size uses up all available RAM, or causes a stack overflow.
[...] Have you tried the latest release, which modified the Map and [a] instances?
Tried right now. My [UArr (Word32 :*: Word8)], where the list length is 17770, now requires 660 MB of memory, when decoding, against 840 MB with the previous version of the binary package. This is fantastic, thanks! Regards Manlio

Grzegorz Chrupala
Hi all, Is there a serialization library other than the Data.Binary from hackage?
There are Iteratees[1]. They're still grok-in-process___ community-wise They are (afaik) currently only used for file input, though it's certainly possible to enumerate a data structure and have an iteratee do the output. Hmmm. Output-iteratees are a very, very, interesting thing to think about. [3] Anyway, they'd almost certainly annihilate all your worries about resource usage. [1] The original: http://okmij.org/ftp/Haskell/Iteratee/ Cabalised and improved (e.g. seek performance, identifier capitalisation): http://inmachina.net/~jwlato/haskell/iteratee/ Monadic Regions[2] might shed some light on Oleg's train of thought regarding all this [2] http://www.cs.rutgers.edu/~ccshan/capability/region-io.pdf [3] Which is just another way of saying "I'd rather shut up now, let me try it first" -- (c) this sig last receiving data processing entity. Inspect headers for copyright history. All rights reserved. Copying, hiring, renting, performance and/or quoting of this signature prohibited.

2009/3/15 Achim Schneider
Hmmm. Output-iteratees are a very, very, interesting thing to think about.
Careful, comments like that have a tendency to invoke Oleg. Next thing you know: Interesting Question --> "Maybe you could..." ^ | | | | v Sound of heads popping <-- Oleg: "Yes you can, *and*..."
participants (7)
-
Achim Schneider
-
ChrisK
-
Don Stewart
-
Grzegorz Chrupala
-
Manlio Perillo
-
porges@porg.es
-
Svein Ove Aas