[Git][ghc/ghc][wip/romes/27514] 2 commits: Rule-based deterministic concurrent downsweep
sheaf pushed to branch wip/romes/27514 at Glasgow Haskell Compiler / GHC Commits: 0a54de07 by sheaf at 2026-10-26T09:09:35+01:00 Rule-based deterministic concurrent downsweep This commit rewrites downsweep as a single query-answering rule (see 'DownsweepRule') that can be executed by concurrent worker threads. The design allows every expensive operation (preprocessing files with CPP, parsing headers, reading interfaces) to be performed concurrently according to the -j<N>/-jsem flags. See Note [Rules-based downsweep] in GHC.Driver.Downsweep. The rules are run by "GHC.Driver.Concurrency.runRules" in a demand-driven way: each worker can demand other workers to run, but never wait on another worker. Every piece of work is done at most once. Fixes #27514 Memory usage can increase due to doing more work concurrently: ------------------------- Metric Increase: MultiComponentModulesRecomp MultiComponentModulesRecomp100 mhu-perf ------------------------- - - - - - 25535b7f by sheaf at 2026-10-26T09:10:12+01:00 Acquire/release semaphore tokens more readily This commit reworks GHC.Driver.MakeSem to use the scoped worker abstraction of GHC.Utils.Concurrent.Scope. This allows us to get rid of a lot of the tricky logic in GHC.Driver.MakeSem involving manual exception handling, thread spawning and lifetime management, etc. The main user-visible change is that token acquisition is no longer rate limited (due to a bug, there previously was a 1s rate limit on acquisition of semaphore tokens), with the release debounce period shortened from 1s to 10ms. Other changes: - With -jsem, the number of capabilities is now capped at the number of processors, bringing it in line with the -jN behaviour. - Pending jobs whose wait for a token is interrupted by an exception are now withdrawn from the jobserver, instead of acquiring a token that it hung onto without doing useful work. Fixes #27763 - - - - - 24 changed files: - + changelog.d/T27763 - + changelog.d/parallel-downsweep - compiler/GHC/Driver/Concurrency.hs - compiler/GHC/Driver/Downsweep.hs - compiler/GHC/Driver/Env.hs - compiler/GHC/Driver/Errors/Ppr.hs - compiler/GHC/Driver/Errors/Types.hs - compiler/GHC/Driver/Make.hs - compiler/GHC/Driver/MakeSem.hs - compiler/GHC/Types/Error/Codes.hs - compiler/GHC/Utils/Concurrent/Scope.hs - testsuite/tests/diagnostic-codes/codes.stdout - testsuite/tests/driver/T27461/Makefile - + testsuite/tests/driver/T27461/T27461c.stderr - testsuite/tests/driver/T27461/all.T - + testsuite/tests/driver/T27461/src/Wrong.hs - testsuite/tests/ghc-api/downsweep/IncrementalDownsweep.hs - testsuite/tests/ghc-api/downsweep/OldModLocation.hs - testsuite/tests/ghc-api/downsweep/PartialDownsweep.hs - testsuite/tests/ghc-api/fixed-nodes/FixedNodes.hs - testsuite/tests/ghc-api/fixed-nodes/InterfaceModuleGraph.hs - testsuite/tests/ghc-api/fixed-nodes/ModuleGraphInvariants.hs - testsuite/tests/splice-imports/SI35.hs - utils/check-ppr/Main.hs The diff was not included because it is too large. View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/154b1aa3f46628153fa23b3d6e11997... -- View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/154b1aa3f46628153fa23b3d6e11997... You're receiving this email because of your account on gitlab.haskell.org. Manage all notifications: https://gitlab.haskell.org/-/profile/notifications | Help: https://gitlab.haskell.org/help
participants (1)
-
sheaf (@sheaf)