Joachim Breitner
Hi,
this is mostly a rant, targeted at the very vague fact that runtime performance benchmarks are so damn hard.
This commit just came in: https://git.haskell.org/ghc.git/commitdiff/56c9bb39246f9ffd8ed41a0656bfe8e60... Clearly, only parts of the profiling RTS have changed.
This is what perf.haskell.org measures: https://perf.haskell.org/ghc/#revision/56c9bb39246f9ffd8ed41a0656bfe8e60d23b... Highlights: nofib/time/cryptarithm1 0.502 + 9.36% 0.549 seconds nofib/time/fasta 0.406 + 4.43% 0.424 seconds These changes are permanent, i.e. the following commits consistently give the higher runtimes.
But nofib does not even run the profiling RTS!
*very audible sigh*
(Conclusion: Ignore runtime changes in nofib when the RTS has changed, the results are simply unusable.)
I also wonder if we can adjust the input and parameters to nofib (in a new “veryslow” setting) to get them all run in a few seconds on modern hardware, instead of subseconds.
I think this would be a very worthwhile effort. Sub-second runtimes really are just plain useless. Cheers, - Ben