Hannes Siebenhandl pushed to branch wip/fendor/pit-copy-small-slices-scratch at Glasgow Haskell Compiler / GHC Commits: f8a1c1af by Hannes Siebenhandl at 2026-11-04T15:33:20+01:00 Lazy copy of byte strings - - - - - ec8dc296 by Hannes Siebenhandl at 2026-11-04T15:33:20+01:00 Binary: fix putFullBinData after dropping fbd_off_e; factor out copyBinReaderSlice "Lazy copy of byte strings" removed the end offset from FullBinData, so fbd_size is now the size of the whole buffer and the slice extends from fbd_off_s to its end. putFullBinData still wrote fbd_size bytes starting at fbd_off_s, i.e. it read past the end of the buffer whenever the slice did not start at 0 (the mi_hi_bytes of every interface read from disk). It now writes fbd_size - fbd_off_s bytes. Also: * factor the copying out of lazyGetCopy' into copyBinReaderSlice, so it can be used for other regions of an interface file; * make fbd_readerUserData strict, and addReaderToUserData strict, so that neither leaves a thunk retaining the old ReadBinHandle (and its buffer); * export makeAbsoluteBin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - f5d93bde by Hannes Siebenhandl at 2026-11-04T15:33:20+01:00 Replace the PIT by slim ExternalIfaces in the EPS The EPS no longer keeps a ModIface per loaded external module (eps_PIT :: ModuleEnv ModIface). Instead, eps_ext_ifaces holds an ExternalIface (new module GHC.Unit.External.Iface) with only what is queried after the declarations, instances, rules etc. went into the EPS pools: identity, exports, fixities (+ env), warnings (+ lookup fns), ABI hashes, the decl-hash env, dependencies, trust and (lazily) docs. It is built strictly from the freshly loaded interface (after plugins ran), so nothing retains the ModIface. See Note [The EPS does not keep ModIfaces]. loadInterface and its wrappers (loadSysInterface, loadSrcInterface, loadInterfaceForName/Module, loadModuleInterface, loadPluginInterface, getModuleInterface, hscGetModuleInterface) now return a LoadedIface: either the HUG's ModIface of a home module, or the ExternalIface from the EPS. Its li_* accessors replace the mi_* accessors at all call sites (renamer, recompilation checking, usages, family instance consistency, Safe Haskell, Backpack, plugins, GHCi, TH, haddock). Fields that are no longer kept: * mi_top_env (TH reifyModule) and, for the GHC API, the full ModIface of package modules (modInfoIface): re-read from disk with the new readFullIfaceFromDisk. * mi_usages, self-recompilation info: never needed for external modules. One-shot mode (POC, marked "TODO: one-shot PIT access"): * home-unit interfaces still land in the EPS, now as ExternalIfaces; * readFixedModule no longer consults the EPS but always reads the .hi; * haddock's loadHiFile re-reads the full interface from disk. Behavioural change: a plugin's interfaceLoadAction result is what the EPS remembers (previously only the first loadInterface caller saw it). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - 4619aeb2 by Hannes Siebenhandl at 2026-11-04T15:33:20+01:00 Don't retain .hi buffers from the EPS Even without the PIT, lazily deserialised parts of external interfaces retained the buffer of the whole .hi file: * The IfaceType deduplication table was read from the file buffer; its entries are lazy thunks holding that buffer, and every lazily read part of the interface reaches the table via the ReaderUserData. getTables now copies the bytes of the IfaceType table into a buffer of its own (its extent is known from the table layout written by putAllTables). See Note [Copying the IfaceType table]. * The lazily read (type, details, idinfo) of an IfaceId held the buffer of the whole interface until forced; most Ids in the EPS are never forced. It now uses lazyGetCopy, so an unforced Id only retains its own bytes. * The bytecode-loading action stored in eps_iface_bytecode and iface_core_bindings captured the whole ModIface via lazy selectors of mi_module; force the module instead. Compiling Cabal-syntax (-O1), the byte arrays of at least 16 KiB reachable from the EPS drop from 30.1 MiB (374) to 1.8 MiB (58); the remaining ones are the lazily kept docs and large single Ids (see bench/eps-memory). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - 3d88b788 by Hannes Siebenhandl at 2026-11-04T15:33:20+01:00 bench/eps-memory: ghc-debug benchmark of memory reachable from the EPS Compiles a package (default: Cabal-syntax from ../../cabal, flags via hie-bios) with a frozen ghc-debug-enabled compiler, pauses it at the end (GHC_DEBUG_PAUSE_AT_END) and reports per EPS field the reachable and exclusive sizes, byte arrays and top closure types, plus retainer paths of big byte arrays; eps-compare compares variants. See README.md. OVERVIEW.md summarises this branch and what is still open. Results, frozen compilers and work directories are not committed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - ba4b84fe by fendor at 2026-11-04T15:33:20+01:00 WIP: One-shot mode: home-unit interfaces go to the HPT, never the EPS In one-shot mode (ghc -c), interfaces of home-unit modules used to be loaded into the External Package State (PIT, PTE and the instance / family-instance / rule / annotation / COMPLETE pools). Home modules belong to the home unit, so nothing about them should live in the EPS. With this change the PIT only holds third-party interfaces (except GHC.Prim), so its representation can change freely. * loadInterface: a demand-loaded home interface is typechecked into ModDetails and added to the HPT as a HomeModInfo (without linkable) before any lazy decl thunk is forced. * New GHC.Unit.Home.LoadedState: HomeLoadedState, a per-HPT record (second IORef in HomePackageTable) with the pools of the demand-loaded home modules: hls_inst_env, hls_fam_inst_env, hls_rule_base, hls_ann_env, hls_complete_matches, hls_iface_bytecode, hls_boot_types (own hi-boot hack) and hls_is_boot (moved from eps_is_boot). Note [Demand-loaded home interfaces]. In --make mode it stays empty. * Consumers read EPS pool + home pool live (instances, family instances, rules, annotations, COMPLETE pragmas, bytecode). RuleEnv gets a separate re_loaded_home_rules field, refreshed by override (updLoadedHomeRules) like re_eps_rules, so Specialise writing se_rules back does not duplicate rules. * lookupTypeInPTE no longer short-circuits in one-shot mode; importDecl uses lookupType. get_linkable falls back to the finder for HPT entries without a linkable. * Failed home loads are not memoised (no fake iface in the PIT). * eps_is_boot removed from ExternalPackageState. * Test: testsuite/tests/driver/OneShotHomeHpt (A -> B -> C compiled with -c; instance, type family instance and RULE from C used in A). Verified: devel2+ipe build and devel2+ipe+assertions_stage1 rebuild (stage-2 libraries are compiled in one-shot mode), driver/th/ indexed-types/simplCore/plugins/backpack/typecheck tests. Open follow-ups: GHC.Prim still in the PIT; eps_known_keys may hold home GHC.Essentials entries; eps_free_holes to be folded into a new PIT representation; stale comment in lookupIfaceByModule (case (a)). Keywords: one-shot HPT, HomeLoadedState, hls_, eps_is_boot, PIT only external (cherry picked from commit 4c44e66e05a201029de3fe9ec9dcf295f6e2931d) Conflicts with "Replace the PIT by slim ExternalIfaces in the EPS", resolved: eps_is_boot is dropped next to eps_ext_ifaces; loadInterface returns LoadedHome for demand-loaded home modules (with a trimmed ModIface in the HPT, built from the plugins' result) and builds/forces the ExternalIface only for external modules; failed home loads are not memoised in eps_ext_ifaces. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - 82f7a4c5 by Hannes Siebenhandl at 2026-11-04T15:33:20+01:00 Resolve the "one-shot PIT access" TODOs With demand-loaded home interfaces in the HPT (one-shot mode), no home module is in the EPS any more (except GHC.Prim), so: * lookupIfaceByModule: drop the one-shot case of the comment. * readFixedModule: check the HUG/EPS again before reading the .hi; take the dependencies from the LoadedIface (li_deps), so it also works for ExternalIfaces. * haddock's loadHiFile: take the instances from the ModDetails the module now has in the HPT instead of typechecking them a second time; the declarations still come from re-reading the full interface (neither the HPT entry nor the EPS keep them). No more panic on a failed re-read: report the interface error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - f4ebc5f1 by Hannes Siebenhandl at 2026-11-04T15:33:20+01:00 One-shot: don't count demand-loaded home instances twice in runTcInteractive runTcInteractive seeds tcg_inst_env with the instances of all HPT modules (hugAllInstances). In one-shot mode the HPT contains only demand-loaded home modules, whose instances tcGetInstEnvs/tcGetFamInstEnvs already take from the HomeLoadedState pools, so every instance was reported twice (haddockHtmlTest/haddockLatexTest, --no-compilation runs in OneShot mode). Skip hugAllInstances in one-shot mode and document this in Note [Demand-loaded home interfaces]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - f350d7f2 by Hannes Siebenhandl at 2026-11-04T15:33:20+01:00 bench/eps-memory: update OVERVIEW after the one-shot changes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - 49bbb03d by Hannes Siebenhandl at 2026-11-04T15:33:20+01:00 Remove the GHC.Prim special case for demand-loaded home interfaces GHC.Internal.Prim's interface is virtual (getGhcPrimIface, substituted by readIface for the .hi file on disk), and it is a home module only while ghc-internal itself is compiled. In --make mode it is an ordinary module of the graph whose HPT entry gets the virtual interface; in one-shot mode it was excluded from the home path and still went into the EPS. Treat it like any other demand-loaded home module: it now goes into the HPT (empty ModDetails, empty pools), so no home module is in the EPS. Also drop the stale "unsavoury" GHC.Prim filter in hugSomeThingsBelowUs (GHC.Prim is compiled, hence in the HPT, in --make mode) and the GHC.Prim case of the lookupIfaceByModule comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - cd3c9321 by Hannes Siebenhandl at 2026-11-04T15:33:20+01:00 haddock: take default-method names from ModDetails, don't re-read .hi files In one-shot and --no-compilation mode, loadHiFile re-read every module's interface from disk, because the interface of a demand-loaded home module in the HPT does not keep its declarations. createInterface1 needed mi_decls only for the Names of default methods; take them from the module's type environment (md_types) instead, which both modes have. loadHiFile now only loads the interface (into the HPT) and uses the HPT entry: the trimmed ModIface, md_types and the typechecked instances. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - 30dfc1d8 by Hannes Siebenhandl at 2026-11-04T15:33:20+01:00 Add ifaceReadHook; changelog for the LoadedIface API change GHC no longer keeps the full interfaces of external modules, so GHC API users need another way to get hold of them. The new observe-only hook ifaceReadHook :: Maybe (FilePath -> ModIface -> IO ()) is called by readIface with the full interface every time an interface file is read from disk (EPS loads, one-shot HPT loads, recompilation checks, fixed module nodes). Changing interfaces stays the job of plugins' interfaceLoadAction. Test: ghc-api/IfaceReadHook. Changelog: changelog.d/slim-eps-api lists the functions that now return a LoadedIface. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - 031aa4f6 by Hannes Siebenhandl at 2026-11-04T15:33:20+01:00 bench/eps-memory: update OVERVIEW (GHC.Prim, haddock, ifaceReadHook) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - b7c34beb by Hannes Siebenhandl at 2026-11-04T15:33:20+01:00 bench/eps-memory: add run-peak.sh / eps-peak for heap profiles of the peak Compiles once with -ddump-timings and +RTS -l -hT -i0.25, and analyses the eventlog: peak live bytes after a major GC, the GHC phase and module running at that time, per-module maxima and the -hT census nearest to the peak; eps-peak compare diffs two variants. RTS max_bytes_used only samples the ~20 major GCs of a build and misses the real peak. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - 9fc8c500 by Hannes Siebenhandl at 2026-11-04T15:33:20+01:00 bench/eps-memory: absolute numbers in comparisons, MiB and MB, all peak measures Comparisons now show the absolute value of every variant, with the change relative to the baseline in percent; sizes are given in MiB and, in brackets, MB (eventlog2html's unit). eps-peak reports the max live bytes over all GCs, after major GCs, the largest -hT census and the max heap size. OVERVIEW: peak table and corrected peak observation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - 33416f43 by Hannes Siebenhandl at 2026-11-04T15:33:20+01:00 bench/eps-memory: measure pinned-block fragmentation eps-analyse now also writes PREFIX-frag.{md,hsdat}: a census of the pinned block groups at the end-of-build pause (single 4 KiB pinned blocks, which hold many small pinned objects, vs multi-block large objects), with live bytes reachable from the saved roots, utilisation histogram, wasted bytes and the sizes of the live objects. eps-peak additionally reports the memory in blocks vs. live bytes at the peak and the largest gap at a major GC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - 81 changed files: - + bench/eps-memory/.gitignore - + bench/eps-memory/OVERVIEW.md - + bench/eps-memory/README.md - + bench/eps-memory/app/Analyse.hs - + bench/eps-memory/app/Compare.hs - + bench/eps-memory/app/Flags.hs - + bench/eps-memory/app/Peak.hs - + bench/eps-memory/app/Results.hs - + bench/eps-memory/cabal.project - + bench/eps-memory/eps-bench.cabal - + bench/eps-memory/freeze-compiler.sh - + bench/eps-memory/run-bench.sh - + bench/eps-memory/run-peak.sh - + changelog.d/slim-eps-api - compiler/GHC/Core/Opt/Monad.hs - compiler/GHC/Core/Opt/Pipeline.hs - compiler/GHC/Core/Opt/Simplify.hs - compiler/GHC/Core/Opt/Specialise.hs - compiler/GHC/Core/Rules.hs - compiler/GHC/Driver/Downsweep.hs - compiler/GHC/Driver/Env.hs - compiler/GHC/Driver/Hooks.hs - compiler/GHC/Driver/Main/Compile.hs - compiler/GHC/Driver/Main/Hsc.hs - compiler/GHC/Driver/Main/Interactive.hs - compiler/GHC/Driver/Main/Passes.hs - compiler/GHC/Driver/Plugins.hs - compiler/GHC/Driver/Session/Inspect.hs - compiler/GHC/HsToCore.hs - compiler/GHC/HsToCore/Monad.hs - compiler/GHC/HsToCore/Usage.hs - compiler/GHC/Iface/Binary.hs - compiler/GHC/Iface/Load.hs - compiler/GHC/Iface/Load.hs-boot - compiler/GHC/Iface/Make.hs - compiler/GHC/Iface/Recomp.hs - compiler/GHC/Iface/Rename.hs - compiler/GHC/Iface/Syntax.hs - compiler/GHC/IfaceToCore.hs - compiler/GHC/Linker/Deps.hs - compiler/GHC/Linker/Loader.hs - compiler/GHC/Rename/Env.hs - compiler/GHC/Rename/Fixity.hs - compiler/GHC/Rename/Names.hs - compiler/GHC/Rename/Utils.hs - compiler/GHC/Runtime/Eval.hs - compiler/GHC/Runtime/Loader.hs - compiler/GHC/Tc/Deriv/Utils.hs - compiler/GHC/Tc/Errors/Hole.hs - compiler/GHC/Tc/Errors/Ppr.hs - compiler/GHC/Tc/Errors/Types.hs - compiler/GHC/Tc/Gen/Splice.hs - compiler/GHC/Tc/Instance/Family.hs - compiler/GHC/Tc/Module.hs - compiler/GHC/Tc/Utils/Backpack.hs - compiler/GHC/Tc/Utils/Env.hs - compiler/GHC/Tc/Utils/Instantiate.hs - compiler/GHC/Tc/Utils/Monad.hs - compiler/GHC/Types/GREInfo.hs - compiler/GHC/Unit/External.hs - + compiler/GHC/Unit/External/Iface.hs - compiler/GHC/Unit/Home/Graph.hs - + compiler/GHC/Unit/Home/LoadedState.hs - compiler/GHC/Unit/Home/PackageTable.hs - compiler/GHC/Unit/Module/Deps.hs - compiler/GHC/Unit/Module/ModIface.hs - compiler/GHC/Utils/Binary.hs - compiler/ghc.cabal.in - ghc/Main.hs - + testsuite/tests/driver/OneShotHomeHpt/A.hs - + testsuite/tests/driver/OneShotHomeHpt/B.hs - + testsuite/tests/driver/OneShotHomeHpt/C.hs - + testsuite/tests/driver/OneShotHomeHpt/Makefile - + testsuite/tests/driver/OneShotHomeHpt/OneShotHomeHpt.stdout - + testsuite/tests/driver/OneShotHomeHpt/all.T - + testsuite/tests/ghc-api/IfaceReadHook.hs - + testsuite/tests/ghc-api/IfaceReadHook.stdout - + testsuite/tests/ghc-api/IfaceReadHook_A.hs - testsuite/tests/ghc-api/all.T - utils/haddock/haddock-api/src/Haddock/Interface.hs - utils/haddock/haddock-api/src/Haddock/Interface/Create.hs The diff was not included because it is too large. View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/106aa673352d9e6ea53621c6a622122... -- View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/106aa673352d9e6ea53621c6a622122... You're receiving this email because of your account on gitlab.haskell.org. Manage all notifications: https://gitlab.haskell.org/-/profile/notifications | Help: https://gitlab.haskell.org/help