Re: [Haskell-cafe] ST not strict enough?

2011/11/15 Roman Cheplyaka :

* Jason Dusek [2011-11-15 20:08:48+0000]

...

I'm having some trouble with memory usage in rebuilding a ByteString with some sequences escaped. I thought I'd try vectors. However, it seems that even a relatively simple function, one that places all the bytes of a ByteString in to a vector, uses a great deal of memory.

I think what's happening here is ByteString's "strictness" makes things actually lazy on your side.

Namely, unpack function produces its result "strictly", whole list at once. As a result, the resulting list cannot be consumed one-by-one, so it takes memory. You see ST thunks because

 mapM_ f as =  sequence_ (map f as)

and that map probably gets fused with unpack.

I guess the proper solution here is to use lazy bytestring and make sure the chunks are not very big.

Hi Roman, Switching to the lazy ByteStrings API does, indeed, help; total memory usage is around 16M. I will have a look at the rules that are fired to see what I can learn. -- Jason Dusek ()  ascii ribbon campaign - against html e-mail /\  www.asciiribbon.org   - against proprietary attachments