Re: [Haskell-cafe] 1G strings in Haskell

dagit:

On 4/19/06, Donald Bruce Stewart wrote:

...

Question: Can I manipulate 1G strings in Haskell?

Failed due to memory exhaustion. Almost made it though, just need a tad more ram than I had.

filter ! unlines ! unwords ! reverse ! -- copy cons ! -- copy snoc ! -- involves a copy ++ ! -- can't concat two 1G strings on this box

I would say given the nature of the experiment that you want to feed these functions with values that lead to 1 GB results. At least, that makes sense in the case of ++, but maybe not the others.

Yes, good point. Previous experiments with 500M strings indicated that these all worked fine: Size of test data: 531416k Fine: filter 7.264 unwords 1.715 reverse 1.980 cons 0.976 snoc 0.689 ++ 2.845 Still wouldn't work at 0.5G unlines ! concat !

Did the machine have any available swap? If so, why was it not used?

Swap was used, 1G more, but ulimits kicked in a few times (and I'm not an admin on the box). This was a bit frustrating, but the 0.5G results show what is possible.

The strings had to be in memory all at once to not skew the benchmark by including I/O time?

The strings were all in memory, yes, with some deepSeq tricks. -- Don