Sebastian Graf pushed to branch wip/sg/enter-data at Glasgow Haskell Compiler / GHC Commits: c62264d2 by Sebastian Graf at 2026-08-03T13:02:40+02:00 StgToCmm: evaluate in tail position through stg_enter_data emitEnter's Return case jumped to the closure's entry code unconditionally, entering pointers that are forced already, unlike the sibling AssignTo case which tests the tag inline. Implement the design from #27594: test the tag in a shared RTS stub and jump to it, which also unblocks #23173. stg_enter_data is stg_ap_0_fast specialised to the domain getCallMethod reaches with EnterIt, lifted pointers whose type rules out functions: the FUN/PAP/BCO arms are unreachable there and an indirection's entry code follows the indirection, so the closure-type load and the switch on it are dead. emitEnter's Return case becomes a single jump. Measured on nofib imaginary+spectral, both sides built with -fproc-alignment=64 to work around non-deterministic loop header alignment effects: allocations unchanged on all 74 benchmarks, instructions +0.07% geometric mean, compile time unchanged. Instruction count is the wrong metric here. The tag test mostly fails, because tag inference already routes statically known pointers to InferedReturnIt, so what reaches EnterIt is mostly genuine thunks where the test is an extra jump and branch. Where it succeeds it replaces a load feeding an indirect jump, which cycle counts see and instruction counts do not: primetest -1.5% and exact-reals -1.0% cycles at flat instruction counts, against fft2 +0.9% and fft +0.4%. Emitting the test at each call site instead costs a temporary, two basic blocks and an explicit return per site, and makes the compiler allocate 3% more on T13960. - - - - - 4 changed files: - compiler/GHC/StgToCmm/Expr.hs - rts/Apply.cmm - rts/RtsSymbols.c - rts/include/stg/MiscClosures.h Changes: ===================================== compiler/GHC/StgToCmm/Expr.hs ===================================== @@ -36,6 +36,8 @@ import GHC.Cmm.BlockId import GHC.Cmm hiding ( succ ) import GHC.Cmm.Info import GHC.Cmm.Utils ( cmmTagMask, mkWordCLit ) +import GHC.Cmm.CLabel ( mkCmmCodeLabel ) +import GHC.Unit ( rtsUnitId ) import GHC.Platform.Tag ( mAX_PTR_TAG ) import GHC.Core import GHC.Core.DataCon @@ -1190,22 +1192,14 @@ emitEnter fun = do ; adjustHpBackwards ; sequel <- getSequel ; updfr_off <- getUpdFrameOff - ; align_check <- stgToCmmAlignCheck <$> getStgToCmmConfig ; case sequel of - -- For a return, we have the option of generating a tag-test or - -- not. If the value is tagged, we can return directly, which - -- is quicker than entering the value. This is a code - -- size/speed trade-off: when optimising for speed rather than - -- size we could generate the tag test. - -- - -- Right now, we do what the old codegen did, and omit the tag - -- test, just generating an enter. + -- For a return we jump to stg_enter_data, which returns an already + -- tagged pointer to our caller and enters an untagged one. + -- See Note [Forcing an unknown data pointer] in rts/Apply.cmm. Return -> do - { let entry = entryCode platform - $ closureInfoPtr platform align_check - $ CmmReg (nodeReg platform) - ; emit $ mkJump profile NativeNodeCall entry - [cmmUntag platform fun] updfr_off + { let enter_data = CmmLit (CmmLabel + (mkCmmCodeLabel rtsUnitId (fsLit "stg_enter_data"))) + ; emit $ mkJump profile NativeNodeCall enter_data [fun] updfr_off ; return AssignedDirectly } ===================================== rts/Apply.cmm ===================================== @@ -47,6 +47,31 @@ import CLOSURE stg_apply_interp_info; import CLOSURE stg_restore_cccs_eval_info; #endif +/* Note [Forcing an unknown data pointer] + * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + * stg_enter_data forces a pointer to a thunk, a constructor, or an + * indirection to one, and returns the result to the caller's continuation. + * emitEnter (GHC.StgToCmm.Expr) jumps here for the pointers getCallMethod + * classifies as EnterIt: lifted ones whose type rules out functions, so + * whether they are forced already is unknown until the tag is tested. + * + * The stub is stg_ap_0_fast specialised to that domain, where its + * FUN/PAP/BCO arms are unreachable and an indirection's own entry code + * follows the indirection. What remains is the tag test and an entry jump. + * + * Whether the pointer turns out to be tagged decides which way a program moves: + * nofib imaginary+spectral is a wash overall, with primetest and exact-reals a + * percent faster and fft2 and fft a percent slower. + */ + +stg_enter_data ( P_ x ) +{ + if (GETTAG(x) != 0) { + return (x); + } + jump %GET_ENTRY(x) (x); +} + /* ---------------------------------------------------------------------------- * Evaluate a closure and return it. * ===================================== rts/RtsSymbols.c ===================================== @@ -842,6 +842,7 @@ extern char **environ; SymI_HasDataProto(stg_ap_ppppp_info) \ SymI_HasDataProto(stg_ap_pppppp_info) \ SymI_HasDataProto(stg_ap_0_fast) \ + SymI_HasDataProto(stg_enter_data) \ SymI_HasDataProto(stg_ap_v_fast) \ SymI_HasDataProto(stg_ap_f_fast) \ SymI_HasDataProto(stg_ap_d_fast) \ ===================================== rts/include/stg/MiscClosures.h ===================================== @@ -300,6 +300,7 @@ RTS_RET(stg_ap_ppppp); RTS_RET(stg_ap_pppppp); RTS_FUN_DECL(stg_ap_0_fast); +RTS_FUN_DECL(stg_enter_data); RTS_FUN_DECL(stg_ap_v_fast); RTS_FUN_DECL(stg_ap_f_fast); RTS_FUN_DECL(stg_ap_d_fast); View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/commit/c62264d2ad17abb7fb8ce7ca8d91aa0d... -- View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/commit/c62264d2ad17abb7fb8ce7ca8d91aa0d... You're receiving this email because of your account on gitlab.haskell.org. Manage all notifications: https://gitlab.haskell.org/-/profile/notifications | Help: https://gitlab.haskell.org/help