sheaf pushed to branch wip/romes/27514 at Glasgow Haskell Compiler / GHC Commits: 9aacb029 by sheaf at 2026-10-05T18:24:05+02:00 Driver: structured concurrent worker abstraction This commit introduces a structured concurrency framework in the style of the 'ki' library: a collection of threads within a scope. We implement two kinds of concurrent workers on top of this framework: - Independent workers cannot wait for one another at all. The only scheduling operation is to wait for quiescence. - Coordinating workers declare an STM readiness condition (waiting on other workers to complete) which gates their start. See Note [Deterministic concurrent workers] in GHC.Driver.Concurrency. This commit ports upsweep to this new framework, with downsweep being left as subsequent work. Further changes along the way: - Refactoring of how concurrency is acquired to avoid the footgun of trying to use a no-op 'AbstractSem' as a lock in the serial case. - The "re-run with -j1" logic for semaphore opening errors no longer triggers on late semaphore failures (part-way through a lengthy computation). - Logger threads are properly cleaned up on exception, with each concurrent worker's log queue and local TmpFs properly bracketed. - The 'GhcMessage -> AnyGhcDiagnostic' and 'Maybe Messager' arguments of 'depanalE', 'depanalPartial' and 'downsweep', which were all dead in practice, have been dropped. - Concurrent logging has been completely overhauled. Each worker has its own log queue. Workers whose logs are printed as we go are spawned in the order in which their logs must be printed, now using a TQueue instead of the old complicated LogQueueQueue mechanism. 'logThread' is replaced by 'withLogPrinter', which runs the thread that prints the log queues within a concurrent worker scope. These changes allow us to fix issues with the exception handling: we now always flush remaining log queues before exiting (even when the action throws an exception), and exceptions that arise while printing are properly rethrown. - - - - - 3645b1fc by sheaf at 2026-10-05T18:24:05+02:00 Rule-based deterministic concurrent downsweep This commit rewrites downsweep as a single query-answering rule (see 'DownsweepRule') that can be executed by concurrent worker threads. The design allows every expensive operation (preprocessing files with CPP, parsing headers, reading interfaces) to be performed concurrently according to the -j<N>/-jsem flags. See Note [Rules-based downsweep] in GHC.Driver.Downsweep. The rules are run by "GHC.Driver.Concurrency.runRules" in a demand-driven way: each worker can demand other workers to run, but never wait on another worker. Every piece of work is done at most once. Fixes #27514 ------------------------- Metric Increase: MultiComponentModulesRecomp MultiComponentModulesRecomp100 ------------------------- - - - - - f2731bcf by sheaf at 2026-10-05T18:24:06+02:00 Acquire/release semaphore tokens more readily This commit reworks GHC.Driver.MakeSem to use the scoped worker abstraction of GHC.Utils.Concurrent.Scope. This allows us to get rid of a lot of the tricky logic in GHC.Driver.MakeSem involving manual exception handling, thread spawning and lifetime management, etc. The main user-visible change is that token acquisition is no longer rate limited (due to a bug, there previously was a 1s rate limit on acquisition of semaphore tokens), with the release debounce period shortened from 1s to 10ms. Other changes: - With -jsem, the number of capabilities is now capped at the number of processors, bringing it in line with the -jN behaviour. - Pending jobs whose wait for a token is interrupted by an exception are now withdrawn from the jobserver, instead of acquiring a token that it hung onto without doing useful work. Fixes #27763 - - - - - 30 changed files: - + changelog.d/T27763 - + changelog.d/parallel-downsweep - + compiler/GHC/Driver/Concurrency.hs - + compiler/GHC/Driver/Config/Concurrency.hs - compiler/GHC/Driver/Downsweep.hs - compiler/GHC/Driver/Env.hs - compiler/GHC/Driver/Errors/Ppr.hs - compiler/GHC/Driver/Errors/Types.hs - compiler/GHC/Driver/Make.hs - compiler/GHC/Driver/MakeAction.hs - compiler/GHC/Driver/MakeSem.hs - compiler/GHC/Driver/Pipeline/LogQueue.hs - compiler/GHC/Types/Error/Codes.hs - + compiler/GHC/Utils/Concurrent/Scope.hs - compiler/GHC/Utils/TmpFs.hs - compiler/ghc.cabal.in - testsuite/tests/diagnostic-codes/codes.stdout - testsuite/tests/driver/T27461/Makefile - + testsuite/tests/driver/T27461/T27461c.stderr - testsuite/tests/driver/T27461/all.T - + testsuite/tests/driver/T27461/src/Wrong.hs - testsuite/tests/ghc-api/downsweep/IncrementalDownsweep.hs - testsuite/tests/ghc-api/downsweep/OldModLocation.hs - testsuite/tests/ghc-api/downsweep/PartialDownsweep.hs - testsuite/tests/ghc-api/fixed-nodes/FixedNodes.hs - testsuite/tests/ghc-api/fixed-nodes/InterfaceModuleGraph.hs - testsuite/tests/ghc-api/fixed-nodes/ModuleGraphInvariants.hs - testsuite/tests/splice-imports/SI35.hs - utils/check-ppr/Main.hs - utils/haddock/haddock-api/src/Haddock/Interface.hs The diff was not included because it is too large. View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/020fd5d5a985d0bba56b32675a294fb... -- View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/020fd5d5a985d0bba56b32675a294fb... You're receiving this email because of your account on gitlab.haskell.org. Manage all notifications: https://gitlab.haskell.org/-/profile/notifications | Help: https://gitlab.haskell.org/help