Salam Muhaimin,

So to answer my own question, the current practice seems to be to just eyeball

>   Geometric Mean          -0.0%     -0.0%     -0.3%     -0.1%     +0.1%

and if the numbers are within historical epsilons of 0, that means no change.

For a moment, I thought this was some erratically behaving VM.

The compile times are across-the-board lower, nice!

-- Kim-Ee


On Sat, Sep 28, 2013 at 4:39 AM, Muhaimin Ahsan <leroux@fezrev.com> wrote:
Kim-Ee,

The updated fib-analyse report from a few hours ago is posted here: https://gist.github.com/leroux/6725810#file-headvordnub-analysis-L2988

Sorry for the misunderstanding.

Muhaimin

On Sep 27, 2013, at 3:56 PM, Kim-Ee Yeoh <ky3@atamo.com> wrote:


On Fri, Sep 27, 2013 at 8:14 PM, GHC <ghc-devs@haskell.org> wrote:
A 5% improvement in compile time is remarkable, if it's true.  Great!  But
 I'm always worried about the noise in compile times measured in seconds.

Does anyone else think the noise in runtimes is alarming considering that the following is the fib-analysis of /binary-identical/ programs?

>              Min          -0.1%     -0.0%    -25.4%    -32.2%     -1.3%
>             Max          +0.1%     +0.0%    +19.0%    +22.2%    +10.0%

Shouldn't we find an explanation for this before believing the compile time numbers? What would cause these wide swings on the benchmarking machine?

p.s. For the record: Should do more rigorous statistical testing instead of naive percentages, yes?

-- Kim-Ee
_______________________________________________
ghc-tickets mailing list
ghc-tickets@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-tickets


_______________________________________________
ghc-devs mailing list
ghc-devs@haskell.org
http://www.haskell.org/mailman/listinfo/ghc-devs