On Fri, Feb 5, 2010 at 4:31 PM, Valery V. Vorotyntsev wrote:

...

John Lato wrote:

...

Both designs appear to offer similar performance in aggregate, although there are differences for particular functions.  I haven't yet had a chance to test the performance of the CPS variant, although Oleg has indicated he expects it will be higher.

@jwlato: Do you mind creating `IterateeCPS' tree in http://inmachina.net/~jwlato/haskell/iteratee/src/Data/, so we can start writing CPS performance testing code?

I'm working on the CPS version and will make it public when it's done. It may take a week or so; this term started at 90 and has picked up. I have several benchmark sources that aren't public yet, but I can put them online for your perusal.

AFAICS, you have benchmarks for IterateeM-driven code already: http://inmachina.net/~jwlato/haskell/iteratee/tests/benchmarks.hs

Those will make more sense when I've added the context of the codebases in use. There are several more sets of output that I simply haven't published yet, including bytestring-based variants.

John Millikin wrote:

...

I wrote some criterion benchmarks for IterateeM vs IterateeCPS, and the CPS version was notably slower. I don't understand enough about CPS to diagnose why, but the additional runtime was present in even simple cases (reading from a file, writing back out).

That's very interesting. I wonder if I'll see the same, and if I'd be able to figure it out myself... Did you benchmark any cases without doing IO? Sometimes the cost of the IO can overwhelm any other measurable differences, and also disk caching can affect results. Criterion should highlight any major outliers, but I still like to avoid IO when benchmarking unless strictly necessary.

@jmillikin: Could you please publish those benchmarks?

+1 John

I've put my benchmarking code online at: http://inmachina.net/~jwlato/haskell/research-iteratee.tar.bz2 unpack it so you have this directory structure: ./iteratee ./research-iteratee/ Also download my criterionProcessor programs. The darcs repo is at http://inmachina.net/~jwlato/haskell/criterionProcessor/ to use it, go into the criterionProcessor directory, edit the testrunner.hs script for your environment, and run it. This runs all the benchmarks. Then you can use the CritProc program (build with cabal) to generate pictures. I'm pretty sure you need Chart HEAD in order to build CritProc (I hacked my Chart install, but I think the only important change has been applied to HEAD). I make no guarantees that these will all build properly, it's basically a work-in-progress dump. John On Fri, Feb 5, 2010 at 10:25 PM, John Millikin wrote:

Benchmark attached. It just enumerates a list until EOF is reached.

An interesting thing I've noticed is that IterateeMCPS performs better with no optimization, but -O2 gives IterateeM the advantage. Their relative performance depends heavily on the chunk size -- for example, CPS is much faster at chunk size 1, but slower with 100-element chunks.

On Fri, Feb 5, 2010 at 08:56, John Lato wrote:

...

On Fri, Feb 5, 2010 at 4:31 PM, Valery V. Vorotyntsev wrote:

...

...

John Lato wrote:

...

Both designs appear to offer similar performance in aggregate, although there are differences for particular functions.  I haven't yet had a chance to test the performance of the CPS variant, although Oleg has indicated he expects it will be higher.

@jwlato: Do you mind creating `IterateeCPS' tree in http://inmachina.net/~jwlato/haskell/iteratee/src/Data/, so we can start writing CPS performance testing code?

I'm working on the CPS version and will make it public when it's done.  It may take a week or so; this term started at 90 and has picked up. I have several benchmark sources that aren't public yet, but I can put them online for your perusal.

...

AFAICS, you have benchmarks for IterateeM-driven code already: http://inmachina.net/~jwlato/haskell/iteratee/tests/benchmarks.hs

Those will make more sense when I've added the context of the codebases in use.  There are several more sets of output that I simply haven't published yet, including bytestring-based variants.

...

John Millikin wrote:

...

I wrote some criterion benchmarks for IterateeM vs IterateeCPS, and the CPS version was notably slower. I don't understand enough about CPS to diagnose why, but the additional runtime was present in even simple cases (reading from a file, writing back out).

That's very interesting.  I wonder if I'll see the same, and if I'd be able to figure it out myself...

Did you benchmark any cases without doing IO?  Sometimes the cost of the IO can overwhelm any other measurable differences, and also disk caching can affect results.  Criterion should highlight any major outliers, but I still like to avoid IO when benchmarking unless strictly necessary.

...

@jmillikin: Could you please publish those benchmarks?

+1

John