Re: [Haskell-cafe] why does the binary library require so much memory?

31 Jul 2009

      Why don't you use your own instance to serialize IxSet lazily (or
however you would like?)

There's no reason to be constrained to use the [a] instance.

-- Don

jeremy:
...
Hello,
Is there a work-around? This is killer for Happstack. Most Happstack
applications use IxSet, which in turn uses lists to serialize the data
to/from disk.
Also, why doesn't the stuff get freed eventually?
- jeremy
At Fri, 31 Jul 2009 14:27:30 -0700,
Don Stewart wrote:
...
bos:
...
On Fri, Jul 31, 2009 at 1:56 PM, Jeremy Shaw  wrote:
Using encode/decode from Binary seems to permamently increase my
    memory consumption by 60x fold. I am wonder if I am doing something
    wrong, or if this is an issue with Binary.
It's an issue with the Binary instance for lists, which forces the entire spine
of the list too early. This gives you a gigantic structure to hold onto.
This is the current instance
instance Binary a => Binary [a] where
        put l  = put (length l) >> mapM_ put l
        get    = do n <- get :: Get Int
                    getMany n
-- | 'getMany n' get 'n' elements in order, without blowing the stack.
    getMany :: Binary a => Int -> Get [a]
    getMany n = go [] n
     where
        go xs 0 = return $! reverse xs
        go xs i = do x <- get
                     -- we must seq x to avoid stack overflows due to laziness in
                     -- (>>=)
                     x `seq` go (x:xs) (i-1)
It used to be this, though,
xs <- replicateM n get     -- now the elems.
-- Don