5 Apr
2009
5 Apr
'09
5:10 p.m.
Excerpts from Manlio Perillo's message of Sun Apr 05 22:41:57 +0200 2009:
Manlio Perillo ha scritto:
Hi.
I'm having memory problems decoding a big IntMap.
The data structure is:
IntMap (UArr (Word16 :*: Word8))
There are 480189 keys, and a total of 100480507 elements (Netflix Prize). The size of the encoded (and compressed) data is 184 MB.
When I load data from the Netflix Prize data set, total memory usage is 1030 Mb.
It seems there is a problem with tuples, too.
I have a: [(Word16, UArr (Word32 :*:* Word8))]
This eats more memory than it should, since tuples are decoded lazily.
Why not switch to [(Word16 :*: UArr (Word32 :*: Word8))] then? -- Nicolas Pouillard