On Mon, Feb 9, 2009 at 12:49 PM, Ketil Malde wrote:

Reading the binary file is very efficient thanks to Data.Binary. However, output is a different matter. Currently, my code looks something like:

summarize :: Foo -> ByteString summarize f = let f1 = accessor f f2 = expression f : in B.concat [f1,pack "\t",pack (show f2),...]

which isn't particularly elegant, and builds a temporary ByteString that usually only get passed to B.putStrLn. I can suffer the inelegance were it only fast - but this ends up taking the better part of the execution time.

Is building the strict ByteString what takes the most time? If so, you might want to use `writev` to avoid extra copying. Does your data support incremental processing so that you could produce output before all input has been parsed? Cheers, Johan

On Mon, 2009-02-09 at 12:49 +0100, Ketil Malde wrote:

Hi,

I'm currently working on a program that parses a large binary file and produces various textual outputs extracted from it. Simple enough.

But: since we're talking large amounts of data, I'd like to have reasonable performance.

Reading the binary file is very efficient thanks to Data.Binary. However, output is a different matter. Currently, my code looks something like:

Have you considered using Data.Binary to output the data too? It has a pretty efficient underlying monoid for accumulating output data in a buffer. You'd want some wrapper functions over the top to make it a bit nicer for your use case, but it should work and should be quick. It generates a lazy bytestring, but does so with a few large chunks so the IO will still be quick. Duncan

ketil:

Hi,

I'm currently working on a program that parses a large binary file and produces various textual outputs extracted from it. Simple enough.

But: since we're talking large amounts of data, I'd like to have reasonable performance.

Reading the binary file is very efficient thanks to Data.Binary. However, output is a different matter. Currently, my code looks something like:

summarize :: Foo -> ByteString summarize f = let f1 = accessor f f2 = expression f : in B.concat [f1,pack "\t",pack (show f2),...]

which isn't particularly elegant, and builds a temporary ByteString that usually only get passed to B.putStrLn. I can suffer the inelegance were it only fast - but this ends up taking the better part of the execution time.

Why not use Data.Binary for output too? It is rather efficient at output -- using a continuation-like system to fill buffers gradually. -- Don