-
3184d705
by Hannes Siebenhandl at 2026-11-04T15:23:35+01:00
Binary: fix putFullBinData after dropping fbd_off_e; factor out copyBinReaderSlice
"Lazy copy of byte strings" removed the end offset from FullBinData, so
fbd_size is now the size of the whole buffer and the slice extends from
fbd_off_s to its end. putFullBinData still wrote fbd_size bytes starting
at fbd_off_s, i.e. it read past the end of the buffer whenever the slice
did not start at 0 (the mi_hi_bytes of every interface read from disk).
It now writes fbd_size - fbd_off_s bytes.
Also:
* factor the copying out of lazyGetCopy' into copyBinReaderSlice, so it
can be used for other regions of an interface file;
* make fbd_readerUserData strict, and addReaderToUserData strict, so that
neither leaves a thunk retaining the old ReadBinHandle (and its buffer);
* export makeAbsoluteBin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-
cea353d8
by Hannes Siebenhandl at 2026-11-04T15:23:35+01:00
Replace the PIT by slim ExternalIfaces in the EPS
The EPS no longer keeps a ModIface per loaded external module
(eps_PIT :: ModuleEnv ModIface). Instead, eps_ext_ifaces holds an
ExternalIface (new module GHC.Unit.External.Iface) with only what is
queried after the declarations, instances, rules etc. went into the EPS
pools: identity, exports, fixities (+ env), warnings (+ lookup fns), ABI
hashes, the decl-hash env, dependencies, trust and (lazily) docs. It is
built strictly from the freshly loaded interface (after plugins ran), so
nothing retains the ModIface. See Note [The EPS does not keep ModIfaces].
loadInterface and its wrappers (loadSysInterface, loadSrcInterface,
loadInterfaceForName/Module, loadModuleInterface, loadPluginInterface,
getModuleInterface, hscGetModuleInterface) now return a LoadedIface:
either the HUG's ModIface of a home module, or the ExternalIface from the
EPS. Its li_* accessors replace the mi_* accessors at all call sites
(renamer, recompilation checking, usages, family instance consistency,
Safe Haskell, Backpack, plugins, GHCi, TH, haddock).
Fields that are no longer kept:
* mi_top_env (TH reifyModule) and, for the GHC API, the full ModIface of
package modules (modInfoIface): re-read from disk with the new
readFullIfaceFromDisk.
* mi_usages, self-recompilation info: never needed for external modules.
One-shot mode (POC, marked "TODO: one-shot PIT access"):
* home-unit interfaces still land in the EPS, now as ExternalIfaces;
* readFixedModule no longer consults the EPS but always reads the .hi;
* haddock's loadHiFile re-reads the full interface from disk.
Behavioural change: a plugin's interfaceLoadAction result is what the
EPS remembers (previously only the first loadInterface caller saw it).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-
9f9244dd
by Hannes Siebenhandl at 2026-11-04T15:23:35+01:00
Don't retain .hi buffers from the EPS
Even without the PIT, lazily deserialised parts of external interfaces
retained the buffer of the whole .hi file:
* The IfaceType deduplication table was read from the file buffer; its
entries are lazy thunks holding that buffer, and every lazily read part
of the interface reaches the table via the ReaderUserData. getTables now
copies the bytes of the IfaceType table into a buffer of its own (its
extent is known from the table layout written by putAllTables).
See Note [Copying the IfaceType table].
* The lazily read (type, details, idinfo) of an IfaceId held the buffer of
the whole interface until forced; most Ids in the EPS are never forced.
It now uses lazyGetCopy, so an unforced Id only retains its own bytes.
* The bytecode-loading action stored in eps_iface_bytecode and
iface_core_bindings captured the whole ModIface via lazy selectors of
mi_module; force the module instead.
Compiling Cabal-syntax (-O1), the byte arrays of at least 16 KiB reachable
from the EPS drop from 30.1 MiB (374) to 1.8 MiB (58); the remaining ones
are the lazily kept docs and large single Ids (see bench/eps-memory).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-
f5723884
by Hannes Siebenhandl at 2026-11-04T15:23:35+01:00
bench/eps-memory: ghc-debug benchmark of memory reachable from the EPS
Compiles a package (default: Cabal-syntax from ../../cabal, flags via
hie-bios) with a frozen ghc-debug-enabled compiler, pauses it at the end
(GHC_DEBUG_PAUSE_AT_END) and reports per EPS field the reachable and
exclusive sizes, byte arrays and top closure types, plus retainer paths of
big byte arrays; eps-compare compares variants. See README.md.
OVERVIEW.md summarises this branch and what is still open.
Results, frozen compilers and work directories are not committed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-
4cbabdc8
by fendor at 2026-11-04T15:23:35+01:00
WIP: One-shot mode: home-unit interfaces go to the HPT, never the EPS
In one-shot mode (ghc -c), interfaces of home-unit modules used to be
loaded into the External Package State (PIT, PTE and the instance /
family-instance / rule / annotation / COMPLETE pools). Home modules
belong to the home unit, so nothing about them should live in the EPS.
With this change the PIT only holds third-party interfaces (except
GHC.Prim), so its representation can change freely.
* loadInterface: a demand-loaded home interface is typechecked into
ModDetails and added to the HPT as a HomeModInfo (without linkable)
before any lazy decl thunk is forced.
* New GHC.Unit.Home.LoadedState: HomeLoadedState, a per-HPT record
(second IORef in HomePackageTable) with the pools of the demand-loaded
home modules: hls_inst_env, hls_fam_inst_env, hls_rule_base,
hls_ann_env, hls_complete_matches, hls_iface_bytecode, hls_boot_types
(own hi-boot hack) and hls_is_boot (moved from eps_is_boot).
Note [Demand-loaded home interfaces]. In --make mode it stays empty.
* Consumers read EPS pool + home pool live (instances, family instances,
rules, annotations, COMPLETE pragmas, bytecode). RuleEnv gets a
separate re_loaded_home_rules field, refreshed by override
(updLoadedHomeRules) like re_eps_rules, so Specialise writing se_rules
back does not duplicate rules.
* lookupTypeInPTE no longer short-circuits in one-shot mode; importDecl
uses lookupType. get_linkable falls back to the finder for HPT entries
without a linkable.
* Failed home loads are not memoised (no fake iface in the PIT).
* eps_is_boot removed from ExternalPackageState.
* Test: testsuite/tests/driver/OneShotHomeHpt (A -> B -> C compiled with
-c; instance, type family instance and RULE from C used in A).
Verified: devel2+ipe build and devel2+ipe+assertions_stage1 rebuild
(stage-2 libraries are compiled in one-shot mode), driver/th/
indexed-types/simplCore/plugins/backpack/typecheck tests.
Open follow-ups: GHC.Prim still in the PIT; eps_known_keys may hold home
GHC.Essentials entries; eps_free_holes to be folded into a new PIT
representation; stale comment in lookupIfaceByModule (case (a)).
Keywords: one-shot HPT, HomeLoadedState, hls_, eps_is_boot, PIT only external
(cherry picked from commit 4c44e66e05a201029de3fe9ec9dcf295f6e2931d)
Conflicts with "Replace the PIT by slim ExternalIfaces in the EPS",
resolved: eps_is_boot is dropped next to eps_ext_ifaces; loadInterface
returns LoadedHome for demand-loaded home modules (with a trimmed ModIface
in the HPT, built from the plugins' result) and builds/forces the
ExternalIface only for external modules; failed home loads are not
memoised in eps_ext_ifaces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-
de1c7585
by Hannes Siebenhandl at 2026-11-04T15:23:35+01:00
Resolve the "one-shot PIT access" TODOs
With demand-loaded home interfaces in the HPT (one-shot mode), no home
module is in the EPS any more (except GHC.Prim), so:
* lookupIfaceByModule: drop the one-shot case of the comment.
* readFixedModule: check the HUG/EPS again before reading the .hi; take
the dependencies from the LoadedIface (li_deps), so it also works for
ExternalIfaces.
* haddock's loadHiFile: take the instances from the ModDetails the
module now has in the HPT instead of typechecking them a second time;
the declarations still come from re-reading the full interface (neither
the HPT entry nor the EPS keep them). No more panic on a failed re-read:
report the interface error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-
d922f119
by Hannes Siebenhandl at 2026-11-04T15:23:35+01:00
One-shot: don't count demand-loaded home instances twice in runTcInteractive
runTcInteractive seeds tcg_inst_env with the instances of all HPT modules
(hugAllInstances). In one-shot mode the HPT contains only demand-loaded
home modules, whose instances tcGetInstEnvs/tcGetFamInstEnvs already take
from the HomeLoadedState pools, so every instance was reported twice
(haddockHtmlTest/haddockLatexTest, --no-compilation runs in OneShot mode).
Skip hugAllInstances in one-shot mode and document this in
Note [Demand-loaded home interfaces].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-
9248891c
by Hannes Siebenhandl at 2026-11-04T15:23:35+01:00
bench/eps-memory: update OVERVIEW after the one-shot changes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-
7f41d8cf
by Hannes Siebenhandl at 2026-11-04T15:23:35+01:00
Remove the GHC.Prim special case for demand-loaded home interfaces
GHC.Internal.Prim's interface is virtual (getGhcPrimIface, substituted by
readIface for the .hi file on disk), and it is a home module only while
ghc-internal itself is compiled. In --make mode it is an ordinary module
of the graph whose HPT entry gets the virtual interface; in one-shot mode
it was excluded from the home path and still went into the EPS.
Treat it like any other demand-loaded home module: it now goes into the
HPT (empty ModDetails, empty pools), so no home module is in the EPS.
Also drop the stale "unsavoury" GHC.Prim filter in hugSomeThingsBelowUs
(GHC.Prim is compiled, hence in the HPT, in --make mode) and the GHC.Prim
case of the lookupIfaceByModule comment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-
42d68a78
by Hannes Siebenhandl at 2026-11-04T15:23:35+01:00
haddock: take default-method names from ModDetails, don't re-read .hi files
In one-shot and --no-compilation mode, loadHiFile re-read every module's
interface from disk, because the interface of a demand-loaded home module
in the HPT does not keep its declarations. createInterface1 needed
mi_decls only for the Names of default methods; take them from the
module's type environment (md_types) instead, which both modes have.
loadHiFile now only loads the interface (into the HPT) and uses the HPT
entry: the trimmed ModIface, md_types and the typechecked instances.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-
69f4e411
by Hannes Siebenhandl at 2026-11-04T15:23:35+01:00
Add ifaceReadHook; changelog for the LoadedIface API change
GHC no longer keeps the full interfaces of external modules, so GHC API
users need another way to get hold of them. The new observe-only hook
ifaceReadHook :: Maybe (FilePath -> ModIface -> IO ()) is called by
readIface with the full interface every time an interface file is read
from disk (EPS loads, one-shot HPT loads, recompilation checks, fixed
module nodes). Changing interfaces stays the job of plugins'
interfaceLoadAction.
Test: ghc-api/IfaceReadHook. Changelog: changelog.d/slim-eps-api lists
the functions that now return a LoadedIface.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-
d4e5e203
by Hannes Siebenhandl at 2026-11-04T15:23:35+01:00
bench/eps-memory: update OVERVIEW (GHC.Prim, haddock, ifaceReadHook)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-
ce1f9f42
by Hannes Siebenhandl at 2026-11-04T15:23:36+01:00
bench/eps-memory: add run-peak.sh / eps-peak for heap profiles of the peak
Compiles once with -ddump-timings and +RTS -l -hT -i0.25, and analyses the
eventlog: peak live bytes after a major GC, the GHC phase and module
running at that time, per-module maxima and the -hT census nearest to the
peak; eps-peak compare diffs two variants. RTS max_bytes_used only samples
the ~20 major GCs of a build and misses the real peak.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-
7dd56be4
by Hannes Siebenhandl at 2026-11-04T15:23:36+01:00
bench/eps-memory: absolute numbers in comparisons, MiB and MB, all peak measures
Comparisons now show the absolute value of every variant, with the change
relative to the baseline in percent; sizes are given in MiB and, in
brackets, MB (eventlog2html's unit). eps-peak reports the max live bytes
over all GCs, after major GCs, the largest -hT census and the max heap
size. OVERVIEW: peak table and corrected peak observation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-
106aa673
by Hannes Siebenhandl at 2026-11-04T15:23:36+01:00
bench/eps-memory: measure pinned-block fragmentation
eps-analyse now also writes PREFIX-frag.{md,hsdat}: a census of the pinned
block groups at the end-of-build pause (single 4 KiB pinned blocks, which
hold many small pinned objects, vs multi-block large objects), with live
bytes reachable from the saved roots, utilisation histogram, wasted bytes
and the sizes of the live objects. eps-peak additionally reports the memory
in blocks vs. live bytes at the peak and the largest gap at a major GC.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>