On Wed, Dec 5, 2012 at 12:40 AM, Ben Lippmeier <benl@ouroborus.net> wrote:

On 01/12/2012, at 1:42 AM, Simon Peyton-Jones wrote:

> | > While writing a new nofib benchmark today I found myself wondering
> | > whether all the nofib benchmarks are run just before each release,
>

> I think we could do with a GHC Performance Tsar. Especially now that Simon has changed jobs, we need to try even harder to broaden the base of people who help with GHC. It would be amazing to have someone who was willing to:
>
> * Run nofib benchmarks regularly, and publish the results
>
> * Keep baseline figures for GHC 7.6, 7.4, etc so we can keep
> track of regressions
>
> * Investigate regressions to see where they come from; ideally
> propose fixes.
>
> * Extend nofib to contain more representative programs (as Johan is
> currently doing).
>
> That would help keep us on the straight and narrow.

I was running a performance regression buildbot for a while a year ago, but gave it up because I didn't have time to chase down the breakages. At the time we were primarily worried about the asymptotic performance of DPH, and fretting about a few percent absolute performance was too much of a distraction.

However: if someone wants to pick this up then they may get some use out of the code I wrote for it. The dph-buildbot package in the DPH repository should still compile. This package uses http://hackage.haskell.org/package/buildbox-1.5.3.1 which includes code for running tests, collecting the timings, comparing against a baseline, making pretty reports etc. There is then a second package buildbox-tools which has a command line tool for listing the benchmarks that have deviated from the baseline by a particular amount.

Here is an example of a report that dph-buildbot made:

http://log.ouroborus.net/limitingfactor/dph/nightly-20110809_000147.txt

Ben.

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users@haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users