-
5d0ab71a
by fendor at 2026-07-27T17:31:05-04:00
Introduce global unit database cache
As a first step for better sharing of `UnitInfo` across `UnitEnv`,
we introduce a new datatype called `ExternalUnitDatabases`.
It primarily serves as an in-memory representation of *all*
`UnitDatabase`s across `UnitEnv`. This means, if multiple `HomeUnitEnv`s
depend on the same database, one way or another, we make sure that we
don't parse from disk every time.
Instead, we store the in-memory representation in `ExternalUnitDatabases`.
`ExternalUnitDatabaseCache` is the equivalent of `ExternalUnitState` in
the `UnitEnv`. It is a mutable variable wrapping `ExternalUnitDatabases`.
The mutable `ExternalUnitDatabaseCache` is used in `initUnits` to make
sure we don't parse the same unit database multiple times.
Almost by accident, we change the semantics of `initUnits` to honour
modifications to `packageDBFlags`.
The inability to change `packageDBFlags` while also reusing the already
parsed `UnitDatabase`s was reported in #26423 as a bug.
Hence, we think this behaviour change is warranted and acceptable,
especially since it comes with a breaking change to the `initUnits` API.
Add regression test for #26423
Closes #26423
-
6cce494a
by fendor at 2026-07-27T17:31:05-04:00
Introduce UnitIndex for global external unit caching
`UnitInfo`s have been observed to cause a lot of memory usage in #27500.
Especially with multiple home units, as the same (external) units are
processed from scratch, even though most of the time we end up with
exactly the same `UnitInfo`.
We introduce a `UnitEnv` global cache that allows us to store external
unit information that is used across all `HomeUnitEnv`s.
The most important change in this commit is the introduction of the `UnitIndex`.
It stores a global mapping of `UnitId` -> `UnitInfo`, and `initUnits`
always uses the cached `UnitInfo` entry to populate each
`HomeUnitEnv`'s `UnitState`.
This allows us to ensure the following property:
> Each `UnitInfo` should be alive exactly once in GHC.
All `UnitState`s should reference 'UnitInfo's stored in the 'UnitIndex'.
This ensured by calling 'initUnits' with the 'UnitIndex'.
In addition, the `ExternalUnitDatabases` may also hold a reference
to each on-disk representation of `UnitInfo`.
This means, we impose an hard upper bound on the number of `UnitInfo`s
alive in the GHC session:
> The number of alive `UnitInfo`s closure objects must be the
> sum of all loaded unit database times two.
We add performance regression tests that make sure the number of live
`UnitInfo` cannot exceed this threshold.
Closes #27500
-------------------------
Metric Decrease:
MultiComponentModules
MultiComponentModulesRecomp
MultiComponentModulesRecomp100
mhu-perf
LinkableUsage02
-------------------------
These metrics increases are especially notable, as we are not even
sharing anything big but merely the global package database with 50
entries.
It shows how careful sharing of `UnitInfo` can improve memory usage.
We expect this to be much more notable when the whole cabal package
database is shared across multiple home units.
`LinkableUsage02` metric decreases on unreg and i386 platform, only.
---
Technical details
To share the `UnitInfo`s correctly, it is important that we extract
the `WireMap` into the `UnitIndex`. At the moment of writing, `WireMap`
must be globally the same for all `HomeUnitEnv`s.
This is important, as we could otherwise not cache the "fully-resolved"
`UnitInfo`, as we don't change the `UnitId` or `unitAbiHash` when
resolving wired-in units. Thus, there could be ambiguities, when the
`WireMap` is not the same for all `UnitState`s across the `UnitEnv`.
We consider a `UnitInfo` fully-resolved, if wired-in units have been
updated, the `UnitInfo` has been validated and variables in the unit
config, such as `${pkgroot}` have been resolved.
Updating the wired-in units requires the `WireMap` to be globally the
same.
-
f8e3bee9
by Zubin Duggal at 2026-07-27T17:31:49-04:00
testsuite: skip runtime stats tests on debugged compilers
Debugged flavours build the boot libraries without optimisation, so the
runtime numbers do not match the baselines.
-
1e326770
by Zubin Duggal at 2026-07-27T17:31:50-04:00
testsuite: mark #20706 tests fragile rather than broken
Whether the static linux linker issues manifest depends on the host
toolchain.
-
c0b13cbe
by Zubin Duggal at 2026-07-27T17:31:50-04:00
testsuite: exclude libnuma from mostly-static
It needs static system libraries (libnuma.a) that many platforms do not
ship.
Fixes #26914
-
bee1913d
by Alan Zimmerman at 2026-07-28T16:42:29-04:00
EPA: ClsInstDecl with decls as [LHsDecl GhcPs] in GhcPs
Similar to 4fdfe75731e01dad7d7fa474c2703d0d3965afb1, this commit
changes the as-parsed representation of class instance declarations to
[LHsDecl GhcPs], and only separates them by type from the renamer onward.
This also allows us to remove all the AnnSortKey machinery for exact
printing, as it is now no longer needed.
-
72c55eee
by Cheng Shao at 2026-07-28T16:43:11-04:00
hadrian: implement and use writeFileAtomic to fix race condition
This patch implements `writeFileAtomic` in hadrian and change all
invocations of shake non-atomic `writeFile'` to use `writeFileAtomic`,
to avoid multiple hadrian concurrent invocations overwriting the same
in-tree generated file not in the build root directory. Fixes #27536.
Additional notes:
- `writeFileChanged`/`writeFileChangedBS` cannot be made atomic since
it involves reading the file's older version, so their uses are left
alone. It doesn't affect #27536 given their outputs are contained in
the build root directory.
- It's possible to shrink this patch by only making writes outside the
build root directory atomic. But I think it's not worth the effort
for fine grained distinction here, and atomic writes within the
build root directory should also improve robustness of a hadrian
build.
- In the longer term we do want to make a ghc build only generate
files within the build root directory, though that's a lot of work
and outside the scope of this particular bugfix.
Co-authored-by: Codex <codex@openai.com>
-
46d4f963
by Sylvain Henry at 2026-07-29T06:38:40-04:00
RTS: correctly mark slop bytes when shrinking large arrays (#19048)
Correctly mark slop bytes even when profiling is off so that heap census
doesn't traverse garbage-collected closures.
-
4762a8bf
by Simon Jakobi at 2026-07-29T06:39:23-04:00
Add -XLazyFieldAnnotations (GHC proposal 752)
Unbundle the prefix `~` lazy field annotation syntax from StrictData. The
new LazyFieldAnnotations extension controls whether `~` is accepted on
constructor fields. StrictData (and Strict, transitively) imply the new
extension.
See https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0752-lazy-field-annotations.rst.
Closes #24455.
Assisted-by: Claude Opus 4.8
-
0b6dcc84
by Simon Jakobi at 2026-07-29T06:40:04-04:00
testsuite: Relax T24471 residency tolerance
T24471 peak residency fluctuates enough on i386 to cause spurious
failures. Use the standard residency tolerance while retaining the
existing allocation threshold.
See https://gitlab.haskell.org/ghc/ghc/-/work_items/24471#note_682303.
Assisted-by: gpt-5.6-sol via Codex CLI
-
90e95b34
by Cheng Shao at 2026-07-29T06:40:45-04:00
compiler: fix missing top-level procedure labels in cmm dumps
This patch fixes missing top-level procedure labels in some
intermediate Cmm pass dumps. Fixes #27553.
Co-authored-by: Codex <codex@openai.com>
-
360a5946
by sheaf at 2026-07-29T06:41:35-04:00
Add some type-family-heavy performance tests
FamAppCachePerf stress-tests the performance of lookups in the
type family application cache.
T27336 is a minimisation extracted from the reported reproducer.
SimplCastPerf is a measure of coercion growth due to the simplifier
calling mkTransCo without re-optimising the result.
-
b1b1afec
by Rodrigo Mesquita at 2026-07-31T10:11:41+02:00
loopImports: Don't dup ms_uid in summary imports
We were writing the ms_unitid of the mod summary with every single
import of that module
That complicated the code (as though the UnitId in that list could ever
be something else) and also allocates unnecessarily per every mod
import. Very slight allocation decrease measured locally in a few tests:
(MultiComponentModulesRecomp: -0.06%; MultiComponentModulesRecomp100: -0.05%)
Purely a clean up.
-
9936d45f
by Rodrigo Mesquita at 2026-07-31T10:11:41+02:00
downsweep: make control flow simpler and cache correct
This refactor extracts the control flow of downsweep into a single
function `dfsBuild`, which takes care of iteratively expanding and
traversing all nodes of the in-construction module graph necessary to
build a full `ModuleGraph`.
There are three levels of caching going on, all of which are necessary
to make sure we don't do repeated work (notably, NEVER summarise the
same module twice).
1. `dfsBuild` accumulates the final module graph and never revisits the
same node of the module graph. Cache is keyed by the final
`ModuleGraph`s `NodeKey`s.
2. For Module A in home-unit u1, each import in the list of imports
needs to be *found* (call to `findImportedModuleWithIsBoot`): at this
point, we only have the `ModuleName` of the import, not the `Module`.
This *finding* is somewhat expensive, so we cache it as well
(`ImportsCache`). The cache key is the home-unit to which the module
belongs~[1], the import package qualifier, and the ModuleName.
[1] Different home-units will have different package flags, which means
potentially different `Module` resolution for the same `ModuleName`.
3. The most expensive operation we want to avoid is summarising a
`Module` into a `ModSummary`, which notably involves parsing the
module header from scratch.
The third cache, in essence, maps a `Module` to its `ModSummary`
(named `ModSummaryCache`). This cache upholds the invariant: we NEVER
summarise the same module twice. In practice, the cache key is the
Module's UnitId and the Source path; the reason is we need to
distinguish between `.hs` and `.hs-boot` files, as their summaries
will differ.
Note that (2) can't guarantee this alone: Two ModuleName imports in
separate units can (and likely do) map to the same `Module`.
Note that the previous implementation failed to achieve the
no-duplicate-work summarisation invariant, and we ended up doing a
quadratic amount of processing in scenarios like test
`MultiComponentModules100`.
See also Note [Downsweep Control Flow and Caching]
Fixes #27461
Perf changes:
MultiComponentModules(normal) ghc/alloc 2,097,389,264 1,992,186,736 -5.0% GOOD
MultiComponentModules100(normal) ghc/alloc 24,310,173,770 21,293,867,360 -12.4% GOOD
MultiComponentModulesRecomp(normal) ghc/alloc 602,761,394 498,543,984 -17.3% GOOD
MultiComponentModulesRecomp100(normal) ghc/alloc 11,885,968,240 8,895,404,864 -25.2% GOOD
-------------------------
Metric Decrease:
MultiComponentModules
MultiComponentModules100
MultiComponentModulesRecomp
MultiComponentModulesRecomp100
-------------------------
-
6e6cf839
by Rodrigo Mesquita at 2026-07-31T10:11:41+02:00
implicitRequirementsShallow can never reach HoleUnit
findImportedModule will never return `HoleUnit` for a `ModuleName`
(a `HoleUnit` can only be found as a signature instantiation, never as a
directly *imported* thing)
Therefore, we can drop `[ModuleName]` returned by
`implicitRequirementsShallow`, which makes many things dead code.
Namely, the call to `implicitRequirementsShallow` from
GHC.Driver.Downsweep which was a performance bottleneck (for doing lots
of duplicate work in findImportedModule) is now entirely gone.
Fixes #27053
In an MR with this patch and the downsweep refactor (previous commit), CI says:
MultiComponentModules(normal) ghc/alloc 2,097,396,728 1,943,662,304 -7.3% GOOD
MultiComponentModules100(normal) ghc/alloc 24,310,182,136 17,227,574,440 -29.1% GOOD
MultiComponentModulesRecomp(normal) ghc/alloc 602,769,518 449,973,656 -25.3% GOOD
MultiComponentModulesRecomp100(normal) ghc/alloc 11,885,976,408 4,828,894,160 -59.4% GOOD
and the Cabal test (building Cabal with ghc --make) improves in the
total time reported by +RTS -s from 54s to 40s reliably on my machine
with default+profiled_ghc flavour. That's a 25% reduction in total run time!
-------------------------
Metric Decrease:
MultiComponentModules
MultiComponentModules100
MultiComponentModulesRecomp
MultiComponentModulesRecomp100
-------------------------
-
18b9c78e
by Rodrigo Mesquita at 2026-07-31T10:11:41+02:00
downsweep: Cache negative results
When traversing a module graph structure, a uniquely identified node
should always expand to the same thing.
I don't see how visiting the same node which failed to be expanded a
first time would ever successfully expand the second time we try to
expand it (eg. when coming from a different edge to it -- it is still
the same node!). The node expansion is local, based just based on the
node itself, not on the path to get there.
Therefore, this patch removes the weird behavior and commentary of
`dfsBuild` wrt to `Nothing` not being cached and being potentially
expanded a second time around to something different, which was
misleading and, ultimately, incorrect.
Now, we have a `MGRes`, which is more explicit about a node being
Skipped just being a node that is ignored whenever it is found (and that
skip is cached) -- and we may want to do this due to failures or due to
just trying nodes which might not work on purpose, like hs-boots.
We uniformly cache positive and negative results and remove the
assumption that there might be an ordering in which the same node
visited at a later time might be expanded differently.
This makes it possible to traverse the module nodes in parallel without
a change in behavior, since there's no longer a hidden ordering
requirement.
-
e1896988
by Rodrigo Mesquita at 2026-07-31T10:11:41+02:00
Organize and clean-up GHC.Driver.Downsweep
Simply some cosmetic changes, moving definitions around to structure the
module better into its relevant sections
(In go (ns ++ ss), it's not a problem to use ++ because it's a good
producer and we won't have to append fully before processing the next
item in go)
-
32868b8b
by fendor at 2026-07-31T10:16:35+02:00
Fixup: don't add interactive_module to module graph