Hi there!
You might have seen failed or stuck or pending darwin builds. Our CI
builders we got generously donated have ~250GB of disk space (which should
be absolutely adequat for what we do), and macOS BigSur does some odd
reservation of 200GB in /System/Volumes/Data, this is despite automatic
updates being disabled and time machine being disabled.
It used to happen only when the system was expecting an update to be
performed and the 200GB were freed after the update was done. After the
latest update to 11.4, however, it seems to have not freed that space. This
leaves the CI machine with ~50GB for for the system + build tools + gitlab
checkouts and builds, and they frequently run out of space :-/
If someone knows how to prevent the system from doing stupid stuff like
this (my hunch is it's keeping a backup of the system pre-udpate, for
disaster recovery). Please come forward, my google searches haven't
revealed anything useful yet.
I have filed a TSI with Apple (still had a few on my developer account),
but I don't expect them to come back to me before the end of June. Next
week is WWDC, and there will be a massive backlog of issues that queued up
leading up to, and during the WWDC. I've also only had very marginal
success with them resolving issues that were not "you wrote this program
wrong".
If everything fails, maybe the solution is to attach some usbc ssd's to the
macs and have gitlab builds be run dedicatedly on those disks. I'm a bit
concerned about performance but we would have to see.
Any ideas are welcome, please also feel free to hit me up on
libera.chat#ghc, or the haskell foundations slack.
Cheers,
Moritz
