Re: performance testing

Ah yes -- very helpful. So the baseline is always (implicitly) the parent commit, if that commit has performance info in the set of notes. So if I have this situation (time flows down) origin/master: blah wip/xyz: big cool change (that slows things down) attempt1: comment out some of big cool change attempt2: comment out more of big cool change and I do a perf test on attempt1, it's quite likely that the perf test will *pass*, because it's comparing to the previous commit. And then when I do a perf test on attempt2 and see a metric *decrease*, that might just be because I've fixed the perf problem... but I haven't actually made an improvement. (This story is true. Names have been changed to protect the innocent.) Is my understanding accurate here? Sorry for not finding the appropriate wiki page sooner. It felt like I was witnessing non-determinism, but it actually makes sense now. Thanks! Richard

On Mar 11, 2020, at 9:59 AM, David Eichmann wrote:

Richard

The performance tests seem to be causing more confusion than I'd of liked. A baseline is established from previous performance test runs. There is a wiki page that may help: https://gitlab.haskell.org/ghc/ghc/wikis/building/running-tests/performance-.... Let me know if that's not sufficiently helpful.

David E

On 3/10/20 11:05 PM, Richard Eisenberg wrote:

...

Hi all,

I'm very confused by how to do performance testing.

I have a patch that seems to cause a performance regression. So I built the patch locally (`validate` flavor, with `make`) and then reproduced the regression. Good. Then, I make a small change, rebuild, commit, and then test. But now it seems that the baseline has changed. (This took some time to discover after seemingly non-deterministic results!)

So: how can I make several different commits, with different experiments, switch between them at will, all without changing my baseline? I do *not* have the magic "metric decrease" bit in my commit message. Or, at least, I didn't put it there.

Thanks! Richard _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- David Eichmann, Haskell Consultant Well-Typed LLP, http://www.well-typed.com

Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs