Sebastian Graf pushed to branch wip/sg/enter-data at Glasgow Haskell Compiler / GHC Commits: ad8cf2db by Sebastian Graf at 2026-08-03T12:14:11+02:00 StgToCmm: evaluate in tail position through stg_enter_data Evaluating a value of unknown form in tail position jumps to its entry code unconditionally, so a value already in normal form is entered anyway. Enforcing that the entry code of a taggable normal form is unreachable (#23173) requires testing the tag here, as the AssignTo case already does. Add stg_enter_data and jump to it, rather than open-coding the test at every call site. The stub is stg_ap_0_fast specialised to the domain getCallMethod reaches with EnterIt, values whose type rules out functions: the FUN/PAP/BCO arms are unreachable there and an indirection's entry code follows the indirection, so the closure-type load and switch are dead. emitEnter's Return case becomes a single jump. nofib imaginary+spectral, both sides built with -fproc-alignment=64: allocations bit-identical, instructions +0.07% geometric mean, compile time unchanged. Cycles move in both directions by up to 1.5%, depending on whether a program's tail evaluations usually find a value or a thunk. - - - - - 4 changed files: - compiler/GHC/StgToCmm/Expr.hs - rts/Apply.cmm - rts/RtsSymbols.c - rts/include/stg/MiscClosures.h Changes: ===================================== compiler/GHC/StgToCmm/Expr.hs ===================================== @@ -36,6 +36,8 @@ import GHC.Cmm.BlockId import GHC.Cmm hiding ( succ ) import GHC.Cmm.Info import GHC.Cmm.Utils ( cmmTagMask, mkWordCLit ) +import GHC.Cmm.CLabel ( mkCmmCodeLabel ) +import GHC.Unit ( rtsUnitId ) import GHC.Platform.Tag ( mAX_PTR_TAG ) import GHC.Core import GHC.Core.DataCon @@ -1190,22 +1192,16 @@ emitEnter fun = do ; adjustHpBackwards ; sequel <- getSequel ; updfr_off <- getUpdFrameOff - ; align_check <- stgToCmmAlignCheck <$> getStgToCmmConfig ; case sequel of - -- For a return, we have the option of generating a tag-test or - -- not. If the value is tagged, we can return directly, which - -- is quicker than entering the value. This is a code - -- size/speed trade-off: when optimising for speed rather than - -- size we could generate the tag test. - -- - -- Right now, we do what the old codegen did, and omit the tag - -- test, just generating an enter. + -- For a return we jump to stg_enter_data, which returns a tagged + -- (hence evaluated) value to our caller and enters an untagged + -- closure. getCallMethod picks EnterIt only for values whose type + -- rules out functions, which is what that stub is specialised to. + -- See Note [Evaluating a value in tail position] in rts/Apply.cmm. Return -> do - { let entry = entryCode platform - $ closureInfoPtr platform align_check - $ CmmReg (nodeReg platform) - ; emit $ mkJump profile NativeNodeCall entry - [cmmUntag platform fun] updfr_off + { let enter_data = CmmLit (CmmLabel + (mkCmmCodeLabel rtsUnitId (fsLit "stg_enter_data"))) + ; emit $ mkJump profile NativeNodeCall enter_data [fun] updfr_off ; return AssignedDirectly } ===================================== rts/Apply.cmm ===================================== @@ -47,6 +47,55 @@ import CLOSURE stg_apply_interp_info; import CLOSURE stg_restore_cccs_eval_info; #endif +/* Note [Evaluating a value in tail position] + * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + * stg_enter_data evaluates a closure that is a thunk, a constructor, or an + * indirection to one, and returns it to the caller's continuation. This is + * stg_ap_0_fast (below) specialised to that domain: the FUN/PAP/BCO arms of + * the generic evaluation are unreachable there, and an indirection's own + * entry code follows the indirection, so what remains is the tag test and an + * entry jump. emitEnter (GHC.StgToCmm.Expr) jumps here for a tail evaluation, + * which getCallMethod reaches only for values whose type rules out functions. + * + * Three shapes evaluate such a value. Measured on nofib imaginary+spectral + * (74 benchmarks, cachegrind instruction counts, both sides built with + * -fproc-alignment=64 so that code placement is held fixed): + * + * - Enter unconditionally, reading the entry code out of the closure. This + * reaches the entry code of an evaluated value, which the enter-taggable + * invariant rules out. It also costs a load of the info pointer feeding + * an indirect jump, plus the entry code's own indirect jump to the + * continuation, whenever the value is already evaluated. + * - Jump to stg_ap_0_fast. Adds a load of the closure type and a + * bounds-checked switch on it, both dead on this domain: +0.19% + * instructions over this stub. + * - Jump to this stub. +0.07% instructions over entering unconditionally, + * allocations unchanged. The tag test mostly fails, because values whose + * tag is statically known are routed to InferedReturnIt by tag inference + * before they reach here, leaving mostly genuine thunks: fft2 +0.36%, + * fft +0.24%. + * + * Cycles move in both directions and by more than the instruction counts, + * because what the test replaces is a load feeding an indirect jump rather + * than straight-line work. Where the value is usually evaluated, primetest + * runs -1.5% and exact-reals -1.0%; where it is usually a thunk, fft2 runs + * +0.9% and fft +0.4%. + * + * The stub keeps the test out of the code generator's per-site output. Open + * coding it at each call site costs a temporary, two basic blocks and an + * explicit return there, which compiling T13960 pays for at around +3% + * allocations; through the stub the compiler emits one jump, as it did when + * it entered unconditionally. + */ + +stg_enter_data ( P_ x ) +{ + if (GETTAG(x) != 0) { + return (x); + } + jump %GET_ENTRY(x) (x); +} + /* ---------------------------------------------------------------------------- * Evaluate a closure and return it. * ===================================== rts/RtsSymbols.c ===================================== @@ -842,6 +842,7 @@ extern char **environ; SymI_HasDataProto(stg_ap_ppppp_info) \ SymI_HasDataProto(stg_ap_pppppp_info) \ SymI_HasDataProto(stg_ap_0_fast) \ + SymI_HasDataProto(stg_enter_data) \ SymI_HasDataProto(stg_ap_v_fast) \ SymI_HasDataProto(stg_ap_f_fast) \ SymI_HasDataProto(stg_ap_d_fast) \ ===================================== rts/include/stg/MiscClosures.h ===================================== @@ -300,6 +300,7 @@ RTS_RET(stg_ap_ppppp); RTS_RET(stg_ap_pppppp); RTS_FUN_DECL(stg_ap_0_fast); +RTS_FUN_DECL(stg_enter_data); RTS_FUN_DECL(stg_ap_v_fast); RTS_FUN_DECL(stg_ap_f_fast); RTS_FUN_DECL(stg_ap_d_fast); View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/commit/ad8cf2dbfbf2ca108ac166927a9a6650... -- View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/commit/ad8cf2dbfbf2ca108ac166927a9a6650... You're receiving this email because of your account on gitlab.haskell.org. Manage all notifications: https://gitlab.haskell.org/-/profile/notifications | Help: https://gitlab.haskell.org/help