Re: [Haskell-cafe] Compressed Data.Map for more efficient RAM usage?

On Thu, Feb 16, 2012 at 4:29 PM, Johan Tibell wrote:

On Thu, Feb 16, 2012 at 2:03 PM, Antoine Latter wrote:

...

You could have a re-implemented HashMap which would un-pack the payload's ByteString constructor into the leaves of the HashMap type itself.

Then you would save on both the keys and the values.

Hah hah - forget what I said. HashMap may be smaller in terms of the structure of the tree it builds, but it can't be smaller in the size of its keys since the hash is lossy :-/

Note that ByteString has a high per-value overhead due to the internal use of ForeignPtrs and indicies that track offset/size: http://blog.johantibell.com/2011/06/memory-footprints-of-some-common-data.ht...

You could store just the raw GHC ByteArray, which drops the indices. You could then promote to a ByteString to call into standard deserialization libraries. I don't know at what point we descend too far into voodoo-magic.

-- Johan