On 10/04/2014 08:58, Simon Peyton Jones wrote:

Friends

The thread below concerns GHC's performance. I'm writing to ask for your help.

In developing GHC we always run 'validate', which runs a lot of regression tests. A few of those are performance tests, but because we do so frequently, none of these performance tests run for long, and none depend on other packages. And they tend to test something very specific.

What we lack is a sustained effort to track a) The performance of GHC itself b) The performance of GHC-compiled programs

At GHC HQ we *aspire* to track this stuff, but in practice we simply don't. Bug-fixing, portability, etc etc always ends up taking priority.

But it's a pity that we don't. For example, Michal's comment below (that his GHC-compiled program has run 10-20% faster with each release of GHC from 6.12) is fantastic -- but we have no data to back it up, or know whether it's just Michael, or more widely true. I also suspect that sometimes we regress and don't know it. People do report this (e.g. #8971), but it's a bit random. Again, it would be great to have a more systematic way to know. In many cases it might be easy to fix; but we can only fix if we know.

We have the nofib suite, and we take that very seriously, but it is showing its age, uses no advanced features, and I'm not sure how representative it is any more.

Johan Tibell and Bryan O'Sullivan agreed to become GHC's Performance Tsars a year or two ago, with a view to focusing on (b) at least, but they are both extremely busy. And I don't think they even attempt to focus on (a).

So I'm wondering: would anyone like to help here? It would mean * Soliciting and gathering together some more substantial benchmarks * Gathering performance numbers * Yelling quickly if the numbers go bad. * Investigating why (e.g. no one has seriously profiled GHC itself for a while, both space and time. I bet there are improvements to be had there.)

Maybe it could be part of the new buildbot team's work? Certainly it'd make sense to use the same nightly-build infrastructure.

Anyway, I'm advertising that there's an un-met need, and I'd love some help.

Let me second this. In particular, I think we regress on GHC performance regularly, because the perf tests just aren't a good way to prevent regressions. When we have a +/- 10% window, someone can commit a 9% regression without triggering a perf failure, but the next patch to come along with a 1% regression will be unfairly blamed. Furthermore, by the time we get to the perf tests we're nearly done and just want to push and go home, not go back and profile GHC. Yet the perf tests have an important purpose: the idea is to catch the problem when we have the crucial piece of information: the patch that caused the regression. Someone can try to optimise GHC later, but they have to start from scratch without the information about what caused the regressions. Having an automated system to track GHC performance would help a lot with this, I think. Cheers, Simon

Thanks

Simon

| -----Original Message----- | From: Michal J. Gajda [mailto:mjgajda@gmail.com] | Sent: 10 April 2014 07:26 | To: ghc-devs@haskell.org; Sergei Trofimovich; Manuel Chakravarty; Simon | Peyton Jones | Subject: Re: ghc-7.8-rc2 in -O2 mode eats all stack and RAM on pandoc- | ,citeproc / highlighting-kate | | Dear Devs, | | On 04/10/2014 09:09 AM, ghc-devs-request@haskell.org wrote: | > Filed the reproducer as a new ticket: | > https://ghc.haskell.org/trac/ghc/ticket/8980 | > | > [ Looks like highlighting-kate asks to be added to | > compiler performance benchmarks (are there such ones?) | > It tends to stress ghc all the time: | > http://hackage.haskell.org/trac/ghc/ticket/3664 | > ] | Please consider adding hPDB too, if you want to stress the optimizer. | It shows GHC optimizer at its best, with at least 10-20% improvement in | every major version of | the compiler since 6.12. Unfortunately at cost of very long compile | times. | Please let me know if I should submit a driver code for automatic | benchmarks it (it is in hPDB-examples.) Thanks! | >> > SpecConstr is too aggressive: it sometimes blows up the program | badly and we have no good | solution. See Trac #7898, #7068, #7944, #5550, #8836. | And #8960, where GHC runs out of memory. (Only in 7.8.) Should be easy | to reproduce by just `cabal install hPDB`. | >> > I notice that the latter three are actually fixed in 7.8, so worth | trying that. If it | still fails, do add instructions to reproduce to one of the above open | tickets, or make a new one. | >> > | >> > Meanwhile you can use -fno-spec-constr to simply switch it off for | offending modules. That should get you going. | -- | Best regards | Michal

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs