17 Mar
2021
17 Mar
'21
12:01 p.m.
On 17 Mar 2021, at 16:16, Andreas Klebinger
While I fully agree with this. We should *always* want to know if a small syntetic benchmark regresses by a lot. Or in other words we don't want CI to accept such a regression for us ever, but the developer of a patch should need to explicitly ok it.
Otherwise we just slow down a lot of seldom-used code paths by a lot.
Now that isn't really an issue anyway I think. The question is rather is 2% a large enough regression to worry about? 5%? 10%?
You probably want a sliding window anyway. Having N 1.8% regressions in a row can still slow things down a lot. While a 3% regression after a 5% improvement is probably fine. - Merijn