Thanks Moritz for that update. The latest is that currently darwin CI is disabled and the merge train is unblocked (*choo choo*). I am testing Moritz's patches to speed-up CI and will merge them in shortly to get darwin coverage back. Cheers, Matt On Wed, May 19, 2021 at 9:46 AM Moritz Angermann wrote:

Matt has access to the M1 builder in my closet now. The darwin performance issue is mainly there since BigSur, and (afaik) primarily due to the amount of DYLD_LIBRARY_PATH's we pass to GHC invocations. The system linker spends the majority of the time in the kernel stat'ing and getelements (or some similar directory) call for each and every possible path.

Switching to hadrian will cut down the time from ~5hs to ~2hs. At some point we had make builds <90min by just killing all DYLD_LIBRARY_PATH logic we ever had, but that broke bindists.

The CI has time values attached and some summary at the end right now, which highlights time spent in the system and in user mode. This is up to 80% sys, 20% user, and went to something like 20% sys, 80% user after nuking all DYLD_LIBRARY_PATH's, with hadrian it's closer to ~25% sys, 75% user.

Of note, this is mostly due to time spent during the *test-suite*, not the actual build. For the actual build make and hadrian are comparable, though I've seen hadrian to oddly have a much higher variance in how long it takes to *build* ghc, whereas the make build was more consistent.

The test-suite quite notoriously calls GHC *a lot of times*, which makes any linker issue due to DYLD_LIBRARY_PATH (and similar lookups) much worse.

If we would finally split building and testing, we'd see this more clearly I believe. Maybe this is motivation enough for someone to come forward to break build/test into two CI steps?

Cheers, Moritz

On Wed, May 19, 2021 at 4:14 PM Matthew Pickering wrote:

...

Hi all,

The darwin pipelines are gumming up the merge pipeline as they are taking over 4 hours to complete on average.

I am going to disable them - https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5785

Please can someone give me access to one of the M1 builders so I can debug why the tests are taking so long. Once I have fixed the issue then I will enable the pipelines.

Cheers,

Matt