Hi Gwern, thanks for the interesting info. I quite often have processing of CSV file data of about 100M-1G done. Thanks a lot, Nick 2013/2/2 Gwern Branwen

On Sat, Feb 2, 2013 at 3:19 PM, Nick Rudnick wrote:

...

Roughly, I would say the differences in runtime can reach a factor as much as 1:10 at many times -- and so I am curious whether this subject has already been observed or even better discussed elsewhere. I have spoken to somebody, and our only plausible conclusion was that software like web browsers is able to somewhat aggressively claim system resources higher in the privilege hierarchy (cache?? register??), so that they are not available to other programs any more.

Maybe the Haskell program requires a lot of disk IO? That could easily lead to a big performance change since disk is so slow compared to everything else these days. You could try looking with 'lsof' to see if the browser has a ton of files open or try running the Haskell program with higher or lower disk IO priority via 'ionice'.

-- gwern http://www.gwern.net

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

If you are doing row-by-row transformations, I would recommend giving a try to my csv-conduit or csv-enumerator packages on Hackage. They were designed with constant space operation in mind, which may help you here. If you're keeping an accumulator around, however, you may still run into issues with too much laziness. Ozgun On Feb 2, 2013, at 7:17 PM, Erik de Castro Lopo wrote:

Nick Rudnick wrote:

...

thanks for the interesting info. I quite often have processing of CSV file data of about 100M-1G done.

What library are you using to process the CSV? I have had problems with excessive laziness causing processing of a 75Meg CSV file consuming 500+ megabytes and after I fixed it memory usage dropped to under a megabyte. Processing time dropped from over 10 minutes to about 2 minutes.

I blogged my problem and solution here:

http://www.mega-nerd.com/erikd/Blog/CodeHacking/Haskell/my_space_is_leaking....

I probably need to revisit that because the problem can probably be fixed without deepseq-generics and just using BangPatterns.

Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Ozgun Ataman wrote:

If you are doing row-by-row transformations, I would recommend giving a try to my csv-conduit

I was using csv-conduit.

If you're keeping an accumulator around, however, you may still run into issues with too much laziness.

This was the problem which I solved with deepseq-generic. However, I suspect deepseq-generic was a bigger hammer than was actually needed. Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/