-
f6324171
by sheaf at 2026-09-02T16:13:05+02:00
2-phase Cache/Search Finder monad
This commit restructures the finder abstraction by introducing the
'FinderM' monad, which splits module lookup operations into two phases:
- a cache-only phase, performing no filesystem access,
- from the first cache miss onwards, a search action which may access
the filesystem.
This allows consumers to distinguish between quick cached results versus
more expensive filesystem search operations.
-
c3f5bc7f
by sheaf at 2026-09-02T16:13:48+02:00
Separate Home/External finder caches
The caches for home modules and modules from external units have
different lifetimes, needing to be invalidated at different points.
This commit moves the cache for modules from external units into the
'UnitState' type, which keeps the cache correct whenever package flags
or package databases change (setSessionDynFlags, :set -package ...,
Backpack's addUnit).
This also makes it easier to clear the cache of home modules, as they
are now stored separately (obviating the 'isUnitEnvInstalledModule'
check we used to perform).
-
6c5848fa
by sheaf at 2026-09-02T16:13:52+02:00
Add known home modules to the finder cache
The finder looks in the file system for the on-disk source that
corresponds to a Haskell module name. There are two kinds of modules
where we need to bypass this search:
- a file passed directly in the command line, which may not be in any
search path and whose module name may differ from the file name;
- Backpack signatures and modules, which do not exist as files at all.
This commit handles these two situations by adding an immutable
'KnownHomeModules' in the finder cache, which is always consulted first
before doing any search.
This removes the ad-hoc imperative logic which relied on mutating the
finder cache at careful points.
See Note [Known home modules] in GHC.Unit.Finder.Types.
-
81831dae
by sheaf at 2026-09-02T16:20:05+02:00
Make 'UnresolvedImport' into a suitable cache key
The goal of this commit is to make 'UnresolvedImport' suitable for use
as a cache key. Specifically, during downsweep we want to be able to
cache module resolution queries. We want to do this using some kind of
map, e.g. (roughly speaking):
Map UnresolvedImport ResolvedImport
For this to work, 'UnresolvedImport' must contain **precisely** the
input data to each resolution computation. In other words:
1. If something changes that materially affects resolution, it must
be stored in the cache key.
2. If changing some input does not materially affect the outcome,
then it must not be stored in the cache key.
To satisfy (1), we change 'UnitNode' to also keeps track of the home
unit in whose unit state the unit's dependencies were looked up.
For (2), 'ImportResolution' now stores a 'ModuleLookupScope' instead of an
'UnresolvedImportOrigin'. Plugin imports are expressed with a new
'LookupPlugin' scope. The source location of an import, which is
immaterial to the resolution, is stored out of band outside
'UnresolvedImport', in 'ModSummary'.
-
bd5dac9e
by sheaf at 2026-09-02T16:20:05+02:00
Add GHC.Data.Dependent
This commit adds a minimal implementation of dependent sums
('Some', 'DSum') and dependent maps ('DMap').
The implementation is not entirely performance optimal, as there are a
few extraneous allocations compared to the full-blown implementation in
the 'dependent-map' package. For our use case (caching downsweep
computations), this does not matter much.
See GHC.Data.Dependent.
-
336c2271
by sheaf at 2026-09-02T16:20:05+02:00
Driver: structured concurrent worker abstraction
This commits introduces a structured concurrency framework in the style
of the 'ki' library: a collection of threads within a scope.
We implement two kind of concurrent workers on top of this framework:
- Independent workers cannot wait for one another at all. The only
scheduling operation is to wait for quiescence.
- Coordinating workers declare an STM readiness condition (waiting on
other workers to complete) which gates their start.
See Note [Deterministic concurrent workers] in GHC.Driver.Concurrency.
This commit ports upsweep to this new framework, with downsweep being
left as subsequent work.
Further changes along the way:
- Refactoring of how concurrency is acquired to avoid the footgun of
trying to use a no-op 'AbstractSem' as a lock in the serial case.
- The "re-run with -j1" logic for semaphore opening errors no longer
triggers on late semaphore failures (part-way through a lengthy
computation).
- Logger threads are properly cleaned up on exception, with each
concurrent worker's log queue and local TmpFs properly bracketed.
- The 'GhcMessage -> AnyGhcDiagnostic' and 'Maybe Messager'
arguments of 'depanalE', 'depanalPartial' and 'downsweep', which
were all dead in practice, have been dropped.
-
976bd624
by sheaf at 2026-09-02T16:20:05+02:00
Rule-based deterministic concurrent downsweep
This commit rewrites downsweep as a single query-answering rule
(see 'DownsweepRule') that can be executed by concurrent worker threads.
The design allows every expensive operation (preprocessing files with CPP,
parsing headers, reading interfaces) to be performed concurrently
according to the -j<N>/-jsem flags.
See Note [Rules-based downsweep] in GHC.Driver.Downsweep.
The rules are run by "GHC.Driver.Concurrency.runRules" in a demand-driven
way: each worker can demand other workers to run, but never wait on
another worker. Every piece of work is done at most once.
Fixes #27514
-------------------------
Metric Increase:
MultiComponentModulesRecomp
MultiComponentModulesRecomp100
-------------------------
-
aafa95c3
by sheaf at 2026-09-02T16:20:05+02:00
Acquire/release semaphore tokens more readily
This commit reworks GHC.Driver.MakeSem to use the scoped worker
abstraction of GHC.Utils.Concurrent.Scope. This allows us to get rid of
a lot of the tricky logic in GHC.Driver.MakeSem involving manual
exception handling, thread spawning and lifetime management, etc.
The architecture of the jobserver is rethought: instead of a single
acquire thread, there are as many acquirers as there is a demand for
semaphore tokens. No rate limiting for acquisition.
The release debounce period was shortened from 1s to 10ms.
Fixes #27763