ANNOUNCE: StrictBench 0.1 - Benchmarking code through strict evaluation - Haskell-Cafe

newer
Re: Question on rank-N polymorphism

ANNOUNCE: StrictBench 0.1 - Benchmarking code through strict evaluation

Niemeijer, R.A.

8 Jun 2009 8 Jun '09

7:20 a.m.

Hello everyone, In the last month or so, I've found myself using the following snippet a lot: import Control.Parallel.Strategies import Test.BenchPress bench 1 . print . rnf This snippet fully evaluates a value and prints how long it took to do so. I regularly use it to see where the bottlenecks lie in my algorithms. It has the minor annoyance, however, that it prints a lot of information (min, max, mean, median, percentiles) that is all identical, because I only run it once. The reason I only run it once is that I'm typically evaluating a pure value, which means that any subsequent attempts to benchmark the evaluation time will take no time at all, since it has already been evaluated. To solve this, I decided to write a small library to make this process easier and only print the time taken once. The result is StrictBench, which can be found at http://hackage.haskell.org/cgi-bin/hackage-scripts/package/StrictBench. A short example: import Test.StrictBench main = bench [1..10000000 :: Integer] This code would give 2890.625 ms as output. For the rest of the documentation I refer you to the Hackage page. Regards, Remco Niemeijer

Show replies by date

Magnus Therning

8 Jun 8 Jun

10:58 a.m.

New subject: ANNOUNCE: StrictBench 0.1 - Benchmarking code through strict evaluation

On Mon, Jun 8, 2009 at 8:20 AM, Niemeijer, R.A. wrote:

...

Hello everyone,

In the last month or so, I've found myself using the following snippet a lot:

import Control.Parallel.Strategies import Test.BenchPress

bench 1 . print . rnf

This snippet fully evaluates a value and prints how long it took to do so. I regularly use it to see where the bottlenecks lie in my algorithms. It has the minor annoyance, however, that it prints a lot of information (min, max, mean, median, percentiles) that is all identical, because I only run it once. The reason I only run it once is that I'm typically evaluating a pure value, which means that any subsequent attempts to benchmark the evaluation time will take no time at all, since it has already been evaluated.

To solve this, I decided to write a small library to make this process easier and only print the time taken once. The result is StrictBench, which can be found at http://hackage.haskell.org/cgi-bin/hackage-scripts/package/StrictBench.

Is there no way to force repeated evaluation of a pure value? (It'd be nice to be able to perform time measurements on pure code so that it's possible to compare Haskell implementations of algorithms to implementations in other languages, without running into confounding factors.) /M -- Magnus Therning (OpenPGP: 0xAB4DFBA4) magnus＠therning．org Jabber: magnus＠therning．org http://therning.org/magnus identi.ca|twitter: magthe

Martijn van Steenbergen

12:56 p.m.

New subject: ANNOUNCE: StrictBench 0.1 - Benchmarking code through strict evaluation

Magnus Therning wrote:

...

Is there no way to force repeated evaluation of a pure value? (It'd be nice to be able to perform time measurements on pure code so that it's possible to compare Haskell implementations of algorithms to implementations in other languages, without running into confounding factors.)

I'm really curious about this too. My guess is that the answer is "no" because doing so would (among other things) mean a thunk have to be copied first before it is evaluated, to preserve the unevaluated version. And what guarantee is there that values further down the expression haven't been evaluated already? Efficient lazy evaluation is hard; inefficient lazy evalation is even harder. ;-) Martijn.

Magnus Therning

1:12 p.m.

New subject: ANNOUNCE: StrictBench 0.1 - Benchmarking code through strict evaluation

On Mon, Jun 8, 2009 at 1:56 PM, Martijn van Steenbergen wrote:

...

Magnus Therning wrote:

...
Is there no way to force repeated evaluation of a pure value? (It'd be nice to be able to perform time measurements on pure code so that it's possible to compare Haskell implementations of algorithms to implementations in other languages, without running into confounding factors.)

I'm really curious about this too.

My guess is that the answer is "no" because doing so would (among other things) mean a thunk have to be copied first before it is evaluated, to preserve the unevaluated version. And what guarantee is there that values further down the expression haven't been evaluated already? Efficient lazy evaluation is hard; inefficient lazy evalation is even harder. ;-)

Yes, I guessed as much. I was hoping that there might be some way of tricking GHC into being more inefficient though, something like a rollback in evaluation state. /M -- Magnus Therning (OpenPGP: 0xAB4DFBA4) magnus＠therning．org Jabber: magnus＠therning．org http://therning.org/magnus identi.ca|twitter: magthe

Kenneth Hoste

1:29 p.m.

New subject: ANNOUNCE: StrictBench 0.1 - Benchmarking code through strict evaluation

Hi all, On Jun 8, 2009, at 15:12 , Magnus Therning wrote:

...

On Mon, Jun 8, 2009 at 1:56 PM, Martijn van Steenbergen wrote:

...
Magnus Therning wrote:

...
Is there no way to force repeated evaluation of a pure value? (It'd be nice to be able to perform time measurements on pure code so that it's possible to compare Haskell implementations of algorithms to implementations in other languages, without running into confounding factors.)

I'm really curious about this too.

My guess is that the answer is "no" because doing so would (among other things) mean a thunk have to be copied first before it is evaluated, to preserve the unevaluated version. And what guarantee is there that values further down the expression haven't been evaluated already? Efficient lazy evaluation is hard; inefficient lazy evalation is even harder. ;-)

Yes, I guessed as much. I was hoping that there might be some way of tricking GHC into being more inefficient though, something like a rollback in evaluation state.

I've been playing around with MicroBench [1], and I believe there is a way to trick GHC (at least the 6.10.2 version) into being inefficient. Below is a snippet of the code I used to benchmark various implementations of a function. The key is partial function application and the IO monad. Don't ask me why it works, but I believe it does. -- benchmark a given function by applying it n times to the given value benchmark :: (a -> b) -> a -> Int -> IO() benchmark f x n = do r <- mapM (\y -> return $! f y) (replicate n x) performGC return () The performGC might not be 100% necessary, but I see it as a part of the function evaluation (i.e. make the runtime clean up the mess the function made). Of course this assumes performGC to be called before using "benchmark". Note: MicroBench was doing something similar, but was using mapM_ instead, which no longer seems to fool GHC into evaluating the function n times. mapM does seem to work though. K. -- Kenneth Hoste Paris research group - ELIS - Ghent University, Belgium email: kenneth.hoste@elis.ugent.be website: http://www.elis.ugent.be/~kehoste blog: http://boegel.kejo.be

Sebastian Fischer

1:32 p.m.

New subject: ANNOUNCE: StrictBench 0.1 - Benchmarking code through strict evaluation

On Jun 8, 2009, at 2:56 PM, Martijn van Steenbergen wrote:

...

...
Is there no way to force repeated evaluation of a pure value? I'm really curious about this too.

could it be done by wrapping the computation in a function which is repeatedly called and compiling with -fno-full-laziness to prevent the compiler from floating the computation to the top-level? -- Underestimating the novelty of the future is a time-honored tradition. (D.G.)

Sterling Clover

9 Jun 9 Jun

3:29 a.m.

New subject: ANNOUNCE: StrictBench 0.1 - Benchmarking code through strict evaluation

On Jun 8, 2009, at 6:58 AM, Magnus Therning wrote:

...

Is there no way to force repeated evaluation of a pure value? (It'd be nice to be able to perform time measurements on pure code so that it's possible to compare Haskell implementations of algorithms to implementations in other languages, without running into confounding factors.)

This perhaps introduces too much inefficiency, but one trick is to pack the computation into an existential. i.e. calculate :: Floating b => (forall c. Floating c => c) -> b calculate = id This method is used to evaluate the same numeric formula with different rounding modes in ieee-utils. http://hackage.haskell.org/packages/archive/ieee-utils/0.4.0/doc/html/ src/Numeric-IEEE-Monad.html#perturb Cheers, S.

5877

Age (days ago)

5878

Last active (days ago)

List overview

Download

6 comments

6 participants

participants (6)

Kenneth Hoste
Magnus Therning
Martijn van Steenbergen
Niemeijer, R.A.
Sebastian Fischer
Sterling Clover