Hi Andreas,

I similarly benchmark compiler performance by compiling Cabal, but only occasionally. I mostly trust ghc/alloc metrics in CI and check Cabal when I think there's something afoot and/or want to measure runtime, not only allocations.

I'm inclined to think that for my purposes (testing the impact of optimisations) the GHC codebase offers sufficient variety to turn up fundamental regressions, but maybe it makes sense to build some packages from head.hackage to detect regressions like https://gitlab.haskell.org/ghc/ghc/-/issues/19203 earlier. It's all a bit open-ended and I frankly think I wouldn't get done anything if all my patches would have to get to the bottom of all regressions and improvements on the entire head.hackage set. I somewhat trust that users will complain eventually and file a bug report and that our CI efforts mean that compiler performance will improve in the mean.

Although it's probably more of a tooling problem: I simply don't know how to collect the compiler performance metrics for arbitrary cabal packages.
If these metrics would be collected as part of CI, maybe as a nightly or weekly job, it would be easier to get to the bottom of a regression before it manifests in a released GHC version. But it all depends on how easy that would be to set up and how many CI cycles it would burn, and I certainly don't feel like I'm in a position to answer either question.

Cheers,
Sebastian


Am Mi., 20. Jan. 2021 um 15:28 Uhr schrieb Andreas Klebinger <klebinger.andreas@gmx.at>:
Hello Devs,

When I started to work on GHC a few years back the Wiki recommended
using nofib/spectral/simple/Main.hs as
a test case for compiler performance changes. I've been using this ever
since.

"Recently" the cabal-test (compiling cabal-the-library) has become sort
of a default benchmark for GHC performance.
I've used the Cabal test as well and it's probably a better test case
than nofib/spectral/simple/Main.hs.
I've started using both usually using spectral/simple to benchmark
intermediate changes and then looking
at the cabal test for the final patch at the end. So far I have rarely
seen a large
difference between using cabal or spectral/simple. Sometimes the
magnitude of the effect was different
between the two, but I've never seen one regress/improve while the other
didn't.

Since the topic came up recently in a discussion I wonder if others use
similar means to quickly bench ghc changes
and what your experiences were in terms of simpler benchmarks being
representative compared to the cabal test.

Cheers,
Andreas
_______________________________________________
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs