-
5d0ab71a
by fendor at 2026-07-27T17:31:05-04:00
Introduce global unit database cache
As a first step for better sharing of `UnitInfo` across `UnitEnv`,
we introduce a new datatype called `ExternalUnitDatabases`.
It primarily serves as an in-memory representation of *all*
`UnitDatabase`s across `UnitEnv`. This means, if multiple `HomeUnitEnv`s
depend on the same database, one way or another, we make sure that we
don't parse from disk every time.
Instead, we store the in-memory representation in `ExternalUnitDatabases`.
`ExternalUnitDatabaseCache` is the equivalent of `ExternalUnitState` in
the `UnitEnv`. It is a mutable variable wrapping `ExternalUnitDatabases`.
The mutable `ExternalUnitDatabaseCache` is used in `initUnits` to make
sure we don't parse the same unit database multiple times.
Almost by accident, we change the semantics of `initUnits` to honour
modifications to `packageDBFlags`.
The inability to change `packageDBFlags` while also reusing the already
parsed `UnitDatabase`s was reported in #26423 as a bug.
Hence, we think this behaviour change is warranted and acceptable,
especially since it comes with a breaking change to the `initUnits` API.
Add regression test for #26423
Closes #26423
-
6cce494a
by fendor at 2026-07-27T17:31:05-04:00
Introduce UnitIndex for global external unit caching
`UnitInfo`s have been observed to cause a lot of memory usage in #27500.
Especially with multiple home units, as the same (external) units are
processed from scratch, even though most of the time we end up with
exactly the same `UnitInfo`.
We introduce a `UnitEnv` global cache that allows us to store external
unit information that is used across all `HomeUnitEnv`s.
The most important change in this commit is the introduction of the `UnitIndex`.
It stores a global mapping of `UnitId` -> `UnitInfo`, and `initUnits`
always uses the cached `UnitInfo` entry to populate each
`HomeUnitEnv`'s `UnitState`.
This allows us to ensure the following property:
> Each `UnitInfo` should be alive exactly once in GHC.
All `UnitState`s should reference 'UnitInfo's stored in the 'UnitIndex'.
This ensured by calling 'initUnits' with the 'UnitIndex'.
In addition, the `ExternalUnitDatabases` may also hold a reference
to each on-disk representation of `UnitInfo`.
This means, we impose an hard upper bound on the number of `UnitInfo`s
alive in the GHC session:
> The number of alive `UnitInfo`s closure objects must be the
> sum of all loaded unit database times two.
We add performance regression tests that make sure the number of live
`UnitInfo` cannot exceed this threshold.
Closes #27500
-------------------------
Metric Decrease:
MultiComponentModules
MultiComponentModulesRecomp
MultiComponentModulesRecomp100
mhu-perf
LinkableUsage02
-------------------------
These metrics increases are especially notable, as we are not even
sharing anything big but merely the global package database with 50
entries.
It shows how careful sharing of `UnitInfo` can improve memory usage.
We expect this to be much more notable when the whole cabal package
database is shared across multiple home units.
`LinkableUsage02` metric decreases on unreg and i386 platform, only.
---
Technical details
To share the `UnitInfo`s correctly, it is important that we extract
the `WireMap` into the `UnitIndex`. At the moment of writing, `WireMap`
must be globally the same for all `HomeUnitEnv`s.
This is important, as we could otherwise not cache the "fully-resolved"
`UnitInfo`, as we don't change the `UnitId` or `unitAbiHash` when
resolving wired-in units. Thus, there could be ambiguities, when the
`WireMap` is not the same for all `UnitState`s across the `UnitEnv`.
We consider a `UnitInfo` fully-resolved, if wired-in units have been
updated, the `UnitInfo` has been validated and variables in the unit
config, such as `${pkgroot}` have been resolved.
Updating the wired-in units requires the `WireMap` to be globally the
same.
-
f8e3bee9
by Zubin Duggal at 2026-07-27T17:31:49-04:00
testsuite: skip runtime stats tests on debugged compilers
Debugged flavours build the boot libraries without optimisation, so the
runtime numbers do not match the baselines.
-
1e326770
by Zubin Duggal at 2026-07-27T17:31:50-04:00
testsuite: mark #20706 tests fragile rather than broken
Whether the static linux linker issues manifest depends on the host
toolchain.
-
c0b13cbe
by Zubin Duggal at 2026-07-27T17:31:50-04:00
testsuite: exclude libnuma from mostly-static
It needs static system libraries (libnuma.a) that many platforms do not
ship.
Fixes #26914
-
76322136
by Alan Zimmerman at 2026-07-28T10:32:08-04:00
EPA: ClsInstDecl with decls as [LHsDecl GhcPs] in GhcPs
Similar to 4fdfe75731e01dad7d7fa474c2703d0d3965afb1, this commit
changes the as-parsed representation of class instance declarations to
[LHsDecl GhcPs], and only separates them by type from the renamer onward.
This also allows us to remove all the AnnSortKey machinery for exact
printing, as it is now no longer needed.
-
a2e94c6b
by Cheng Shao at 2026-07-28T10:32:09-04:00
hadrian: implement and use writeFileAtomic to fix race condition
This patch implements `writeFileAtomic` in hadrian and change all
invocations of shake non-atomic `writeFile'` to use `writeFileAtomic`,
to avoid multiple hadrian concurrent invocations overwriting the same
in-tree generated file not in the build root directory. Fixes #27536.
Additional notes:
- `writeFileChanged`/`writeFileChangedBS` cannot be made atomic since
it involves reading the file's older version, so their uses are left
alone. It doesn't affect #27536 given their outputs are contained in
the build root directory.
- It's possible to shrink this patch by only making writes outside the
build root directory atomic. But I think it's not worth the effort
for fine grained distinction here, and atomic writes within the
build root directory should also improve robustness of a hadrian
build.
- In the longer term we do want to make a ghc build only generate
files within the build root directory, though that's a lot of work
and outside the scope of this particular bugfix.
Co-authored-by: Codex <codex@openai.com>