-------- Forwarded Message -------- Subject: Re: performance testing Date: Wed, 11 Mar 2020 12:08:52 +0000 From: David Eichmann To: Richard Eisenberg

Ah yes -- very helpful. So the baseline is always (implicitly) the parent commit, if that commit has performance info in the set of notes. So if I have this situation (time flows down)

origin/master: blah wip/xyz: big cool change (that slows things down) attempt1: comment out some of big cool change attempt2: comment out more of big cool change With you so far. and I do a perf test on attempt1, it's quite likely that the perf test will*pass*, because it's comparing to the previous commit. Yes, assuming you actually ran performance tests on the previous commit with a clean working tree (remember we don't record performance metrics if the git tree has changes, though you'll see a warning in the test output in that case). I'd suggest outputting a graph as described in the wiki if you're wondering which commits have recorded performance metrics. And then when I do a perf test on attempt2 and see a metric*decrease*, that might just be because I've fixed the perf problem... but I haven't actually made an improvement.

Right. The decrease is compared to attempt1, but this may not be a decrease compared to earlier commits. Again I'd suggest graphing the data if you want to inspect this yourself. David E -- David Eichmann, Haskell Consultant Well-Typed LLP, http://www.well-typed.com Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England