Claus Reinke ha scritto:
I split the string in lines, then map some functions on each line to parse the data, and finally calling toU, for converting to an UArr.
Just to make sure (code fragments or, better, reduced examples would make it easier to see what the discussion is about): are you forcing the UArr to be constructed before putting it into the Map?
parse handle = contents <- S.hGetContents handle let v = map singleton' $ ratings contents let m = foldl1' (unionWith appendU) v v `seq` return $! m
where -- Build a Map with a single movie rating singleton' :: (Word32, Word8) -> MovieRatings singleton' (id, rate) = singleton (fromIntegral $ id) (singletonU $ pairS (id, rate))
That helps to make things clearer, I think. One issue is the nature of Maps (strict in keys, non-strict in values).
- neither singleton nor unionWith are strict in the Map values, so nothing here forces the evaluation of rate or construction of UArr
But, as I have written, in one of my tests I also tried rnf to force evaluation: rnf v `seq` rnf m `seq` return m where: instance NFData a => NFData (Data.IntMap.IntMap a) where rnf = rnf . Data.IntMap.toList Isn't this sufficient?
[...] - seq on a list only forces the first node of the list ((:),[],_|_), so (v `seq`) isn't likely to help much. Also, you probably do not want to force the whole list of singletons before builing the Map, you want the singletons to be constructed and consumed incrementally.
Right, thanks.
type Rating = Word32 :*: Word8 type MovieRatings = IntMap (UArr Rating) -- UArr from uvector
A standard trick to keep Map values evaluated by construction is to make the availability of keys dependent on their values, eg (singleton key) $! value. That won't help with unionWith and the appendUs, but it should allow the source string references to be dropped early, as the singletons are constructed.
Tried; but, even using union instead of unionWith, the memory grows fast as before. Just to check if the culprit is IntMap, isn't possible to write a test program that build a big IntMap (UArr Int) ? Right now I don't have the time. Manlio