12 May
2006
12 May
'06
11:35 p.m.
bulat.ziganshin:
Hello Donald,
Friday, May 12, 2006, 10:07:47 AM, you wrote:
Last week Duncan Coutts had the bright idea to write Data.ByteString.Lazy, a newtyped [ByteString], which would allow constant space processing of chunks of strict ByteString arrays.
i have somewhat similar idea - implement getContents variany that returns list of LINES in file, represented as [ByteString]
Data.ByteString.Lazy (5M chunks) constant heap, total time 11.44 minutes, 34% cpu
try to use smaller chunks, don't forget that P4 has 512-1024k cache
After tuning the chunk size to the cache size, the filter 'e' test now runs in: 4.52 minutes, 53% cpu. Which I think is pretty spectacular for 10 gigabytes of data. -- Don