On Fri, Sep 27, 2013 at 8:14 PM, GHC <ghc-devs@haskell.org> wrote:

A 5% improvement in compile time is remarkable, if it's true. Great! But
I'm always worried about the noise in compile times measured in seconds.

Does anyone else think the noise in runtimes is alarming considering that the following is the fib-analysis of /binary-identical/ programs?

> Min -0.1% -0.0% -25.4% -32.2% -1.3%
> Max +0.1% +0.0% +19.0% +22.2% +10.0%

Shouldn't we find an explanation for this before believing the compile time numbers? What would cause these wide swings on the benchmarking machine?

p.s. For the record: Should do more rigorous statistical testing instead of naive percentages, yes?

-- Kim-Ee