bulat.ziganshin:
Hello Donald,
Friday, May 12, 2006, 10:07:47 AM, you wrote:
Last week Duncan Coutts had the bright idea to write Data.ByteString.Lazy, a newtyped [ByteString], which would allow constant space processing of chunks of strict ByteString arrays.
i have somewhat similar idea - implement getContents variany that returns list of LINES in file, represented as [ByteString]
Data.ByteString.Lazy (5M chunks) constant heap, total time 11.44 minutes, 34% cpu
try to use smaller chunks, don't forget that P4 has 512-1024k cache
In fact, here's a graph of time versus chunk size, for my Pentium M laptop, with a 256k L2 cache: http://www.cse.unsw.edu.au/~dons/tmp/chunksize_v_cache.png Lesson: chunksize should be between 0.5 and 1x the L2 cache, and things get much worse as soon as you go beyond your cache size. -- Don