Joachim Breitner
Does anyone have a solid idea what is causing these differences? Are they specific to the builder for perf.haskell.org, or do you observe them as well? And what can we do here?
There is certainly no shortage of possible causes: https://www.youtube.com/watch?v=IX16gcX4vDQ It would be interesting to take a few days to really try to build an understanding of a few of these performance jumps with perf. At the moment we can only speculate.
For the measurements in my thesis I switched to measuring instruction counts (using valgrind) instead. These are much more stable, requires only a single NoFibRun, and the machine does not have to be otherwise quiet. Should I start using these on perf.haskell.org? Or would we lose too much by not tracking actual running times any more?
Note that valgrind can also do cache modelling so I suspect it can give you a reasonably good picture of execution; certainly better than runtime. However, the trade-off is that (last I checked) it's incredibly slow. Don't you think I mean just a bit less peppy than usual. I mean soul-crushingly, mollasses-on-a-cold-December-morning slow. If we think that we can bear the cost of running valigrind then I think it would be a great improvement. As you point out, the current run times are essentially useless. Cheers, - Ben