Re: CI: Choice of base commit for perf comparisons

It seems to be that this thought is in the air right now. This was done just a few days ago: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/7184 https://gitlab.haskell.org/ghc/ghc/-/merge_requests/7231 also looks relevant. Richard

On Dec 22, 2021, at 7:19 AM, Joachim Breitner wrote:

Hi,

the new (or “new”?) handling of perf numbers, where CI just magically records and compares them, without us having to manually edit the `all.T` files, is a big improvement, thanks!

However, I found the choice of the base commit to compare against unhelpful. Assume master is at commit M, and I start a feature branch and MR with commit A. CI runs, and tells me about a performance regressions, and CI is red. I now fix the issue and push commit B to the branch. CI runs, but it picks A to compare against, and now it is red because of an seemingly unexpected performance improvement!

I would have expected that all CI runs for this MR to compare the performance against the base branch on master, and to look for perf change notices in all commit messages in between.

I see these advantages:

* The reported perf changes correspond to the changes shown on the MR page * Green CI = the MR is ready (after squashing) * CI will have numbers for the base commit more reliably (else, if I push commit C quickly after B, then the job for B might be cancelled and Ci will report changes of C against A instead of B, which is unexpected).

I have used this logic of reporting perf changes (or any other “differential CI”) against the base branch in the Motoko project and it was quite natural.

Would it be desirable and possible for us here, too?

(A possible rebuttal might be: we don’t push new commits to feature branches, but always squash and rebase, as that’s what we have to do before merging anyways. If that’s the case then ok, although I generally lean to having chronological commits on feature branches and a nice squashed commit on master.)

Cheers, Joachim

-- Joachim Breitner mail@joachim-breitner.de http://www.joachim-breitner.de/

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs