Sven Tennie pushed to branch wip/supersven/hadrian-cross-stage3 at Glasgow Haskell Compiler / GHC Commits: 5d0ab71a by fendor at 2026-07-27T17:31:05-04:00 Introduce global unit database cache As a first step for better sharing of `UnitInfo` across `UnitEnv`, we introduce a new datatype called `ExternalUnitDatabases`. It primarily serves as an in-memory representation of *all* `UnitDatabase`s across `UnitEnv`. This means, if multiple `HomeUnitEnv`s depend on the same database, one way or another, we make sure that we don't parse from disk every time. Instead, we store the in-memory representation in `ExternalUnitDatabases`. `ExternalUnitDatabaseCache` is the equivalent of `ExternalUnitState` in the `UnitEnv`. It is a mutable variable wrapping `ExternalUnitDatabases`. The mutable `ExternalUnitDatabaseCache` is used in `initUnits` to make sure we don't parse the same unit database multiple times. Almost by accident, we change the semantics of `initUnits` to honour modifications to `packageDBFlags`. The inability to change `packageDBFlags` while also reusing the already parsed `UnitDatabase`s was reported in #26423 as a bug. Hence, we think this behaviour change is warranted and acceptable, especially since it comes with a breaking change to the `initUnits` API. Add regression test for #26423 Closes #26423 - - - - - 6cce494a by fendor at 2026-07-27T17:31:05-04:00 Introduce UnitIndex for global external unit caching `UnitInfo`s have been observed to cause a lot of memory usage in #27500. Especially with multiple home units, as the same (external) units are processed from scratch, even though most of the time we end up with exactly the same `UnitInfo`. We introduce a `UnitEnv` global cache that allows us to store external unit information that is used across all `HomeUnitEnv`s. The most important change in this commit is the introduction of the `UnitIndex`. It stores a global mapping of `UnitId` -> `UnitInfo`, and `initUnits` always uses the cached `UnitInfo` entry to populate each `HomeUnitEnv`'s `UnitState`. This allows us to ensure the following property:
Each `UnitInfo` should be alive exactly once in GHC.
All `UnitState`s should reference 'UnitInfo's stored in the 'UnitIndex'. This ensured by calling 'initUnits' with the 'UnitIndex'. In addition, the `ExternalUnitDatabases` may also hold a reference to each on-disk representation of `UnitInfo`. This means, we impose an hard upper bound on the number of `UnitInfo`s alive in the GHC session:
The number of alive `UnitInfo`s closure objects must be the sum of all loaded unit database times two.
We add performance regression tests that make sure the number of live `UnitInfo` cannot exceed this threshold. Closes #27500 ------------------------- Metric Decrease: MultiComponentModules MultiComponentModulesRecomp MultiComponentModulesRecomp100 mhu-perf LinkableUsage02 ------------------------- These metrics increases are especially notable, as we are not even sharing anything big but merely the global package database with 50 entries. It shows how careful sharing of `UnitInfo` can improve memory usage. We expect this to be much more notable when the whole cabal package database is shared across multiple home units. `LinkableUsage02` metric decreases on unreg and i386 platform, only. --- Technical details To share the `UnitInfo`s correctly, it is important that we extract the `WireMap` into the `UnitIndex`. At the moment of writing, `WireMap` must be globally the same for all `HomeUnitEnv`s. This is important, as we could otherwise not cache the "fully-resolved" `UnitInfo`, as we don't change the `UnitId` or `unitAbiHash` when resolving wired-in units. Thus, there could be ambiguities, when the `WireMap` is not the same for all `UnitState`s across the `UnitEnv`. We consider a `UnitInfo` fully-resolved, if wired-in units have been updated, the `UnitInfo` has been validated and variables in the unit config, such as `${pkgroot}` have been resolved. Updating the wired-in units requires the `WireMap` to be globally the same. - - - - - f8e3bee9 by Zubin Duggal at 2026-07-27T17:31:49-04:00 testsuite: skip runtime stats tests on debugged compilers Debugged flavours build the boot libraries without optimisation, so the runtime numbers do not match the baselines. - - - - - 1e326770 by Zubin Duggal at 2026-07-27T17:31:50-04:00 testsuite: mark #20706 tests fragile rather than broken Whether the static linux linker issues manifest depends on the host toolchain. - - - - - c0b13cbe by Zubin Duggal at 2026-07-27T17:31:50-04:00 testsuite: exclude libnuma from mostly-static It needs static system libraries (libnuma.a) that many platforms do not ship. Fixes #26914 - - - - - bee1913d by Alan Zimmerman at 2026-07-28T16:42:29-04:00 EPA: ClsInstDecl with decls as [LHsDecl GhcPs] in GhcPs Similar to 4fdfe75731e01dad7d7fa474c2703d0d3965afb1, this commit changes the as-parsed representation of class instance declarations to [LHsDecl GhcPs], and only separates them by type from the renamer onward. This also allows us to remove all the AnnSortKey machinery for exact printing, as it is now no longer needed. - - - - - 72c55eee by Cheng Shao at 2026-07-28T16:43:11-04:00 hadrian: implement and use writeFileAtomic to fix race condition This patch implements `writeFileAtomic` in hadrian and change all invocations of shake non-atomic `writeFile'` to use `writeFileAtomic`, to avoid multiple hadrian concurrent invocations overwriting the same in-tree generated file not in the build root directory. Fixes #27536. Additional notes: - `writeFileChanged`/`writeFileChangedBS` cannot be made atomic since it involves reading the file's older version, so their uses are left alone. It doesn't affect #27536 given their outputs are contained in the build root directory. - It's possible to shrink this patch by only making writes outside the build root directory atomic. But I think it's not worth the effort for fine grained distinction here, and atomic writes within the build root directory should also improve robustness of a hadrian build. - In the longer term we do want to make a ghc build only generate files within the build root directory, though that's a lot of work and outside the scope of this particular bugfix. Co-authored-by: Codex <codex@openai.com> - - - - - 46d4f963 by Sylvain Henry at 2026-07-29T06:38:40-04:00 RTS: correctly mark slop bytes when shrinking large arrays (#19048) Correctly mark slop bytes even when profiling is off so that heap census doesn't traverse garbage-collected closures. - - - - - 4762a8bf by Simon Jakobi at 2026-07-29T06:39:23-04:00 Add -XLazyFieldAnnotations (GHC proposal 752) Unbundle the prefix `~` lazy field annotation syntax from StrictData. The new LazyFieldAnnotations extension controls whether `~` is accepted on constructor fields. StrictData (and Strict, transitively) imply the new extension. See https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0752-la.... Closes #24455. Assisted-by: Claude Opus 4.8 - - - - - 0b6dcc84 by Simon Jakobi at 2026-07-29T06:40:04-04:00 testsuite: Relax T24471 residency tolerance T24471 peak residency fluctuates enough on i386 to cause spurious failures. Use the standard residency tolerance while retaining the existing allocation threshold. See https://gitlab.haskell.org/ghc/ghc/-/work_items/24471#note_682303. Assisted-by: gpt-5.6-sol via Codex CLI - - - - - 90e95b34 by Cheng Shao at 2026-07-29T06:40:45-04:00 compiler: fix missing top-level procedure labels in cmm dumps This patch fixes missing top-level procedure labels in some intermediate Cmm pass dumps. Fixes #27553. Co-authored-by: Codex <codex@openai.com> - - - - - 360a5946 by sheaf at 2026-07-29T06:41:35-04:00 Add some type-family-heavy performance tests FamAppCachePerf stress-tests the performance of lookups in the type family application cache. T27336 is a minimisation extracted from the reported reproducer. SimplCastPerf is a measure of coercion growth due to the simplifier calling mkTransCo without re-optimising the result. - - - - - 7d79915f by Sven Tennie at 2026-07-30T08:29:23+02:00 hadrian: Add stage3 cross-bindist target with separate output folder Add a 'binary-dist-stage3' target that packages target executables (produced by the stage2 cross-compiler) into a separate 'bindist-stage3/' folder, distinct from the stage2 cross-compiler bindist in 'bindist/'. Key changes: - BindistConfig: add 'bindistFolder' field so each config knows its parent output folder. crossBindist uses 'bindist', targetBindist uses 'bindist-stage3'. implicitBindistConfig now considers finalStage to pick targetBindist for stage3 builds. - BinaryDist: - Drive configure/Makefile/install-file rules for both folders via forM_ instead of a single 'bindist' path; per-stage distrib directory avoids autoreconf races. - binary-dist-stage3 phony outputs to 'bindist-stage3'. - generateSettings is passed the compiler stage explicitly; ghc-pkg recache uses the executable_stage builder. - pkgToWrappers honors the supplied stage for program naming. - Generate: per-stage configure.ac templates (templateRuleForStages) with correct Build/Host platform interpolation for stage3 (target = host = build). generateSettings takes the compiler stage as a parameter rather than inferring it as predStage. - CabalReinstall: use executable_stage from implicitBindistConfig instead of hard-coded Stage2 for wrapper generation. This allows a single cross-compile to produce both a cross-compiler bindist (x86_64 host) and a target-architecture bindist (e.g. RISC-V) that can be installed and run natively on the target. - - - - - 4b478a5f by Sven Tennie at 2026-07-30T08:29:23+02:00 ci: Combine cross stage2 and stage3 bindists in one job Rename crossConfig to stage2CrossConfig for clarity and turn the nightly x86_64-linux-deb13-riscv-cross job into a combined stage3 job (crossStage = Just 3). With CROSS_STAGE=3, build_hadrian invokes both the 'binary-dist' and 'binary-dist-stage3' targets, then renames the resulting tarballs via BIN_DIST_NAME and BIN_DIST_NAME_STAGE3. The combined job produces two artifacts: the cross-compiler bindist (stage2, runs on x86_64) and the target bindist (stage3, native RISC-V executables) under their historical names, so downstream consumers need no changes. gen_ci.hs: - stage2CrossConfig replaces crossConfig. - binDistNameStage3 generates the backward-compatible stage3 artifact name for jobs with crossStage == Just 3. - jobVariables advertises BIN_DIST_NAME_STAGE3 for stage3 jobs. - stage3Artifacts adds the stage3 tarball to artifactPaths. - cross_jobs: riscv job now sets crossStage = Just 3. ci.sh: - CROSS_STAGE=3 builds both binary-dist and binary-dist-stage3. - After building, rename the stage3 tarball from _build/bindist-stage3/ to $BIN_DIST_NAME_STAGE3.tar.xz. - cross_prefix is empty for CROSS_STAGE=3 (target GHC runs natively on the target, no host-target distinction at install time). jobs.yaml: regenerated. The riscv job keeps its existing name, now with CROSS_STAGE=3 and the extra stage3 artifact + variable. - - - - - 6267f048 by Sven Tennie at 2026-07-30T08:30:21+02:00 Fix: Use library stage for some settings - - - - - 58298583 by Sven Tennie at 2026-07-30T08:30:39+02:00 CI: Use final cross stage wording more consistently - - - - - 6ce80f83 by Sven Tennie at 2026-07-30T08:30:45+02:00 Fix typo - - - - - f6a690a1 by Sven Tennie at 2026-07-30T08:30:50+02:00 Fix wrapper - use executable stage - - - - - 4f7e178d by Sven Tennie at 2026-07-30T08:30:56+02:00 Cleanup - - - - - 68514b9b by Sven Tennie at 2026-07-30T08:44:14+02:00 Add changelog - - - - - 146 changed files: - .gitignore - .gitlab/ci.sh - .gitlab/generate-ci/gen_ci.hs - .gitlab/jobs.yaml - + changelog.d/T26423 - + changelog.d/fix-cmm-dump-labels - + changelog.d/fix-heap-census-large-arrays-19048 - + changelog.d/lazy-field-annotations - + changelog.d/stage3-cross-bindists - + changelog.d/unit-index - compiler/GHC.hs - compiler/GHC/Cmm/Pipeline.hs - compiler/GHC/Driver/Backpack.hs - compiler/GHC/Driver/Env.hs - compiler/GHC/Driver/Flags.hs - compiler/GHC/Driver/Main/Hsc.hs - compiler/GHC/Driver/Main/Interactive.hs - compiler/GHC/Driver/Main/Passes.hs - compiler/GHC/Driver/Session/Units.hs - compiler/GHC/Hs/Decls.hs - compiler/GHC/Hs/Instances.hs - compiler/GHC/Hs/Stats.hs - compiler/GHC/Hs/Utils.hs - compiler/GHC/HsToCore/Docs.hs - compiler/GHC/HsToCore/Quote.hs - compiler/GHC/Iface/Ext/Ast.hs - compiler/GHC/Parser.y - compiler/GHC/Parser/Annotation.hs - compiler/GHC/Parser/PostProcess.hs - compiler/GHC/Rename/Module.hs - compiler/GHC/Rename/Names.hs - compiler/GHC/Tc/Errors/Ppr.hs - compiler/GHC/Tc/Errors/Types.hs - compiler/GHC/Tc/TyCl.hs - compiler/GHC/Tc/TyCl/Class.hs - compiler/GHC/Tc/TyCl/Instance.hs - compiler/GHC/Tc/Utils/Env.hs - compiler/GHC/ThToHs.hs - compiler/GHC/Types/Unique.hs - compiler/GHC/Unit/Env.hs - + compiler/GHC/Unit/External/Database.hs - + compiler/GHC/Unit/External/Index.hs - + compiler/GHC/Unit/External/ModuleOrigin.hs - + compiler/GHC/Unit/External/Providers.hs - + compiler/GHC/Unit/External/Query.hs - + compiler/GHC/Unit/External/Substitution.hs - + compiler/GHC/Unit/External/Validate.hs - + compiler/GHC/Unit/External/Visibility.hs - + compiler/GHC/Unit/External/Wired.hs - compiler/GHC/Unit/Home/Graph.hs - compiler/GHC/Unit/Info.hs - compiler/GHC/Unit/State.hs - compiler/GHC/Unit/State.hs-boot - compiler/GHC/Unit/Types.hs - compiler/Language/Haskell/Syntax/Decls.hs - compiler/ghc.cabal.in - distrib/configure.ac.in - docs/users_guide/exts/strict.rst - ghc/GHCi/UI.hs - hadrian/src/BindistConfig.hs - hadrian/src/Hadrian/Builder/Ar.hs - hadrian/src/Hadrian/Utilities.hs - hadrian/src/Rules/BinaryDist.hs - hadrian/src/Rules/CabalReinstall.hs - hadrian/src/Rules/Documentation.hs - hadrian/src/Rules/Generate.hs - hadrian/src/Rules/Library.hs - hadrian/src/Rules/Nofib.hs - hadrian/src/Rules/Program.hs - hadrian/src/Rules/Register.hs - hadrian/src/Rules/Rts.hs - hadrian/src/Rules/SourceDist.hs - hadrian/src/Rules/Test.hs - hadrian/src/Rules/ToolArgs.hs - libraries/ghc-internal/src/GHC/Internal/LanguageExtensions.hs - rts/Apply.cmm - rts/ZeroSlop.c → rts/MarkSlop.c - rts/PrimOps.cmm - rts/Printer.c - rts/ProfHeap.c - rts/RtsFlags.c - rts/ThreadPaused.c - rts/include/Cmm.h - rts/include/rts/storage/ClosureMacros.h - rts/rts.cabal - rts/sm/NonMovingMark.c - rts/sm/Sanity.c - rts/sm/Storage.c - testsuite/driver/testlib.py - testsuite/tests/count-deps/CountDepsParser.stdout - + testsuite/tests/deSugar/should_run/LazyFieldAnnotationsSemantics.hs - + testsuite/tests/deSugar/should_run/LazyFieldAnnotationsSemantics.stdout - testsuite/tests/deSugar/should_run/all.T - + testsuite/tests/driver/T26423/Hello.hs - + testsuite/tests/driver/T26423/Makefile - + testsuite/tests/driver/T26423/T26423.hs - + testsuite/tests/driver/T26423/T26423.stderr - + testsuite/tests/driver/T26423/T26423.stdout - + testsuite/tests/driver/T26423/all.T - + testsuite/tests/driver/T26423/test/Test.hs - + testsuite/tests/driver/T26423/test/test.pkg - testsuite/tests/driver/T4437.hs - + testsuite/tests/driver/TUnitInfo/Foo.hs - + testsuite/tests/driver/TUnitInfo/Makefile - + testsuite/tests/driver/TUnitInfo/all.T - + testsuite/tests/driver/TUnitInfo/genMhu.sh - + testsuite/tests/driver/TUnitInfo/generic-unit-info-space-mhu.stdout - + testsuite/tests/driver/TUnitInfo/generic-unit-info-space-single.stdout - + testsuite/tests/driver/TUnitInfo/generic-unit-info-space.hs - + testsuite/tests/driver/TUnitInfo/generic-unit-info-space.stdout - testsuite/tests/driver/all.T - testsuite/tests/driver/mostly-static/Makefile - testsuite/tests/ghci/T13786/all.T - testsuite/tests/ghci/linking/all.T - testsuite/tests/ghci/linking/dyn/all.T - testsuite/tests/haddock/should_compile_flag_haddock/T17544.stderr - testsuite/tests/interface-stability/template-haskell-exports.stdout - testsuite/tests/package/T20010/all.T - testsuite/tests/parser/should_compile/DumpRenamedAst.stderr - + testsuite/tests/perf/compiler/FamAppCachePerf.hs - + testsuite/tests/perf/compiler/SimplCastPerf.hs - + testsuite/tests/perf/compiler/T27336.hs - testsuite/tests/perf/compiler/all.T - testsuite/tests/plugins/all.T - testsuite/tests/printer/Test24533.stdout - + testsuite/tests/rts/T19048.hs - + testsuite/tests/rts/T19048.stdout - testsuite/tests/rts/all.T - testsuite/tests/rts/linker/all.T - + testsuite/tests/typecheck/should_compile/LazyFieldAnnotations.hs - testsuite/tests/typecheck/should_compile/all.T - testsuite/tests/typecheck/should_fail/LazyFieldsDisabled.stderr - + testsuite/tests/typecheck/should_fail/LazyFieldsDisabledStrictData.hs - + testsuite/tests/typecheck/should_fail/LazyFieldsDisabledStrictData.stderr - testsuite/tests/typecheck/should_fail/all.T - utils/check-exact/ExactPrint.hs - utils/check-exact/Utils.hs - utils/haddock/haddock-api/src/Haddock.hs - utils/haddock/haddock-api/src/Haddock/Backends/Hoogle.hs - utils/haddock/haddock-api/src/Haddock/Backends/LaTeX.hs - utils/haddock/haddock-api/src/Haddock/Backends/Xhtml/Decl.hs - utils/haddock/haddock-api/src/Haddock/Convert.hs - utils/haddock/haddock-api/src/Haddock/GhcUtils.hs - utils/haddock/haddock-api/src/Haddock/Interface/Create.hs - utils/haddock/haddock-api/src/Haddock/Interface/Rename.hs - utils/haddock/haddock-api/src/Haddock/Types.hs The diff was not included because it is too large. View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/205065115a0536a5cf1ff7d697cbcae... -- View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/205065115a0536a5cf1ff7d697cbcae... You're receiving this email because of your account on gitlab.haskell.org. Manage all notifications: https://gitlab.haskell.org/-/profile/notifications | Help: https://gitlab.haskell.org/help