Matt has access to the M1 builder in my closet now. The darwin performance issue
is mainly there since BigSur, and (afaik) primarily due to the amount of DYLD_LIBRARY_PATH's
we pass to GHC invocations. The system linker spends the majority of the time in the
kernel stat'ing and getelements (or some similar directory) call for each and every possible
path.

Switching to hadrian will cut down the time from ~5hs to ~2hs. At some point we had make
builds <90min by just killing all DYLD_LIBRARY_PATH logic we ever had, but that broke
bindists.

The CI has time values attached and some summary at the end right now, which highlights
time spent in the system and in user mode. This is up to 80% sys, 20% user, and went to
something like 20% sys, 80% user after nuking all DYLD_LIBRARY_PATH's, with hadrian it's
closer to ~25% sys, 75% user.

Of note, this is mostly due to time spent during the *test-suite*, not the actual build. For the
actual build make and hadrian are comparable, though I've seen hadrian to oddly have a
much higher variance in how long it takes to *build* ghc, whereas the make build was more
consistent.

The test-suite quite notoriously calls GHC *a lot of times*, which makes any linker issue due
to DYLD_LIBRARY_PATH (and similar lookups) much worse.

If we would finally split building and testing, we'd see this more clearly I believe. Maybe this
is motivation enough for someone to come forward to break build/test into two CI steps?

Cheers,
 Moritz

On Wed, May 19, 2021 at 4:14 PM Matthew Pickering <matthewtpickering@gmail.com> wrote:
Hi all,

The darwin pipelines are gumming up the merge pipeline as they are
taking over 4 hours to complete on average.

I am going to disable them -
https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5785

Please can someone give me access to one of the M1 builders so I can
debug why the tests are taking so long. Once I have fixed the issue
then I will enable the pipelines.

Cheers,

Matt