long-range performance comparisons (GHC over the years)

Dear Cafe, I recently noticed a performance problem due to a fusion rule not firing for a straightforward piece of code. In fact it turned out this was already fixed in HEAD, see https://ghc.haskell.org/trac/ghc/ticket/11344 https://ghc.haskell.org/trac/ghc/ticket/9848 What worries me is that such a regression had been sitting there for over a year (and did not make it to 7.10.3) and I got to thinking: What long-range performance metering do we have? So I tried to made nofib runs for some ghc-{6,7} versions http://www.imn.htwk-leipzig.de/~waldmann/etc/nofib/comparison-k.text This is slightly broken (not all tests can be built for all compilers, and I don't known how to fix this) but there are some interesting numbers already. It seems nofib programs are self-contained (not using any libraries) so they are mainly using numbers, lists, tuples, and user-defined data. This is the heart of (traditional) Haskell, so this is supposed to work really well. The table shows that there are a lot of benchmarks where performance has been increasing. That's good. But not for all! We should certainly ignore all runtimes that are absolutely small. I think it is most interesting to look allocation numbers. A few examples from this list: * exp3_8 (allocation goes up 50 % from 6.* to 7.*) this is addition of Peano numbers. * gcd (allocation goes up 20 % from 7.8 to 7.10) using Integers, tuples (for extended Euclid), lists (for control) * tak (runtime goes up 20 % from 7.6 to 7.8) the plain Takeuchi function, just Int and recursion (it should not allocate at all?) (and I confirmed these by manually running them for more inputs, all measurements done on debian on x86_64 X5365, ghc-6.* installed from binary packages, ghc-7.* built from source) So, can this be explained? Improved? I think we should resist the temptation to change these benchmarks (using seq and ! and Int# and whatnot) Assuming nofib contains typical code, it is the task of the compiler to handle it well. In case you're wondering about my motivation - this was prompted by teaching, I wanted to show that ghc creates efficient code (by fusion) - but it's not just for the show, I generally try to believe in what I teach and I do rely on this for my real code. (Well, by definition, "real" for me might still be "academic" for others...) - Johannes.

I recently noticed a performance problem
... What worries me is that such a regression
had been sitting there for over a year
There are 215 open runtime performance https://ghc.haskell.org/trac/ghc/query?status=!closed&failure=Runtime+performance+bug&order=id tickets (out of a total of 1655 open tickets, that makes 13%). Compared to say new typesystem features, they don't get much attention. Only a few were fixed https://ghc.haskell.org/trac/ghc/query?failure=Runtime+performance+bug&resolution=fixed&milestone=8.0.1&milestone=7.10.3&milestone=7.10.2 this year, most of them by Joachim Breitner (the new performance tsar https://ghc.haskell.org/trac/ghc/wiki/CodeOwners#CurrentOwners?). He also created https://perf.haskell.org/ghc. In case you want to help out, start here: https://ghc.haskell.org/trac/ghc/wiki/Newcomers.

Yes we’d love help with investigating, characterising, and fixing performance bugs.
Simon
From: Haskell-Cafe [mailto:haskell-cafe-bounces@haskell.org] On Behalf Of Thomas Miedema
Sent: 07 January 2016 00:52
To: Johannes Waldmann

Hi Johannes,
see also this wiki page:
https://ghc.haskell.org/trac/ghc/wiki/Performance/Runtime.
It mentions the `gcd` regression that you found (and has some analysis),
but not the others (`tak` and `exp3_8`). Maybe you could start with
updating that page.
Thomas
On Thu, Jan 7, 2016 at 9:55 AM, Simon Peyton Jones
Yes we’d love help with investigating, characterising, and fixing performance bugs.
Simon
*From:* Haskell-Cafe [mailto:haskell-cafe-bounces@haskell.org] *On Behalf Of *Thomas Miedema *Sent:* 07 January 2016 00:52 *To:* Johannes Waldmann
*Cc:* Joachim Breitner ; Haskell cafe < haskell-cafe@haskell.org> *Subject:* Re: [Haskell-cafe] long-range performance comparisons (GHC over the years) I recently noticed a performance problem
...
What worries me is that such a regression had been sitting there for over a year
There are 215 open *runtime performance https://ghc.haskell.org/trac/ghc/query?status=!closed&failure=Runtime+performance+bug&order=id* tickets (out of a total of 1655 open tickets, that makes 13%).
Compared to say new typesystem features, they don't get much attention. Only a few were fixed https://ghc.haskell.org/trac/ghc/query?failure=Runtime+performance+bug&resolution=fixed&milestone=8.0.1&milestone=7.10.3&milestone=7.10.2 this year, most of them by Joachim Breitner (the new performance tsar https://ghc.haskell.org/trac/ghc/wiki/CodeOwners#CurrentOwners?). He also created https://perf.haskell.org/ghc https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fperf.haskell.org%2fghc%2f&data=01%7c01%7csimonpj%40064d.mgd.microsoft.com%7cc3af688d2c234b3c19b608d316fcb626%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=Ik7nCUHxNalCIxrv9E8dq2od9ed%2bbbLXJ%2fJJyHLCIhQ%3d .
In case you want to help out, start here: https://ghc.haskell.org/trac/ghc/wiki/Newcomers.
participants (3)
-
Johannes Waldmann
-
Simon Peyton Jones
-
Thomas Miedema