Salam Muhaimin, So to answer my own question, the current practice seems to be to just eyeball

Geometric Mean -0.0% -0.0% -0.3% -0.1% +0.1%

and if the numbers are within historical epsilons of 0, that means no change. For a moment, I thought this was some erratically behaving VM. The compile times are across-the-board lower, nice! -- Kim-Ee On Sat, Sep 28, 2013 at 4:39 AM, Muhaimin Ahsan wrote:

Kim-Ee,

The updated fib-analyse report from a few hours ago is posted here: https://gist.github.com/leroux/6725810#file-headvordnub-analysis-L2988 Comment (http://ghc.haskell.org/trac/ghc/ticket/8173#comment:9)

Sorry for the misunderstanding.

Muhaimin

On Sep 27, 2013, at 3:56 PM, Kim-Ee Yeoh wrote:

On Fri, Sep 27, 2013 at 8:14 PM, GHC wrote:

...

A 5% improvement in compile time is remarkable, if it's true. Great! But I'm always worried about the noise in compile times measured in seconds.

Does anyone else think the noise in runtimes is alarming considering that the following is the fib-analysis of /binary-identical/ programs?

...

Min -0.1% -0.0% -25.4% -32.2% -1.3% Max +0.1% +0.0% +19.0% +22.2% +10.0%

Shouldn't we find an explanation for this before believing the compile time numbers? What would cause these wide swings on the benchmarking machine?

p.s. For the record: Should do more rigorous statistical testing instead of naive percentages, yes?

-- Kim-Ee _______________________________________________ ghc-tickets mailing list ghc-tickets@haskell.org http://www.haskell.org/mailman/listinfo/ghc-tickets

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs