GitLab

Sebastian Graf pushed to branch wip/sg/enter-data at Glasgow Haskell Compiler / GHC

Commits:

ad8cf2db

by Sebastian Graf at 2026-08-03T12:14:11+02:00

StgToCmm: evaluate in tail position through stg_enter_data

Evaluating a value of unknown form in tail position jumps to its entry
code unconditionally, so a value already in normal form is entered
anyway. Enforcing that the entry code of a taggable normal form is
unreachable (#23173) requires testing the tag here, as the AssignTo case
already does.

Add stg_enter_data and jump to it, rather than open-coding the test at
every call site. The stub is stg_ap_0_fast specialised to the domain
getCallMethod reaches with EnterIt, values whose type rules out
functions: the FUN/PAP/BCO arms are unreachable there and an
indirection's entry code follows the indirection, so the closure-type
load and switch are dead. emitEnter's Return case becomes a single jump.

nofib imaginary+spectral, both sides built with -fproc-alignment=64:
allocations bit-identical, instructions +0.07% geometric mean, compile
time unchanged. Cycles move in both directions by up to 1.5%, depending
on whether a program's tail evaluations usually find a value or a
thunk.

4 changed files:

compiler/GHC/StgToCmm/Expr.hs
rts/Apply.cmm
rts/RtsSymbols.c
rts/include/stg/MiscClosures.h

Changes:

compiler/GHC/StgToCmm/Expr.hs

@@ -36,6 +36,8 @@ import GHC.Cmm.BlockId
  import GHC.Cmm hiding ( succ )
  import GHC.Cmm.Info
  import GHC.Cmm.Utils ( cmmTagMask, mkWordCLit )
 +import GHC.Cmm.CLabel ( mkCmmCodeLabel )
 +import GHC.Unit ( rtsUnitId )
  import GHC.Platform.Tag ( mAX_PTR_TAG )
  import GHC.Core
  import GHC.Core.DataCon
@@ -1190,22 +1192,16 @@ emitEnter fun = do
    ; adjustHpBackwards
    ; sequel      <- getSequel
    ; updfr_off   <- getUpdFrameOff
 -  ; align_check <- stgToCmmAlignCheck <$> getStgToCmmConfig
    ; case sequel of
 -      -- For a return, we have the option of generating a tag-test or
 -      -- not.  If the value is tagged, we can return directly, which
 -      -- is quicker than entering the value.  This is a code
 -      -- size/speed trade-off: when optimising for speed rather than
 -      -- size we could generate the tag test.
 -      --
 -      -- Right now, we do what the old codegen did, and omit the tag
 -      -- test, just generating an enter.
 +      -- For a return we jump to stg_enter_data, which returns a tagged
 +      -- (hence evaluated) value to our caller and enters an untagged
 +      -- closure. getCallMethod picks EnterIt only for values whose type
 +      -- rules out functions, which is what that stub is specialised to.
 +      -- See Note [Evaluating a value in tail position] in rts/Apply.cmm.
        Return -> do
 -        { let entry = entryCode platform
 -                $ closureInfoPtr platform align_check
 -                $ CmmReg (nodeReg platform)
 -        ; emit $ mkJump profile NativeNodeCall entry
 -                        [cmmUntag platform fun] updfr_off
 +        { let enter_data = CmmLit (CmmLabel
 +                             (mkCmmCodeLabel rtsUnitId (fsLit "stg_enter_data")))
 +        ; emit $ mkJump profile NativeNodeCall enter_data [fun] updfr_off
          ; return AssignedDirectly
+         }

rts/Apply.cmm

@@ -47,6 +47,55 @@ import CLOSURE stg_apply_interp_info;
  import CLOSURE stg_restore_cccs_eval_info;
  #endif
 +/* Note [Evaluating a value in tail position]
 + * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + * stg_enter_data evaluates a closure that is a thunk, a constructor, or an
 + * indirection to one, and returns it to the caller's continuation. This is
 + * stg_ap_0_fast (below) specialised to that domain: the FUN/PAP/BCO arms of
 + * the generic evaluation are unreachable there, and an indirection's own
 + * entry code follows the indirection, so what remains is the tag test and an
 + * entry jump. emitEnter (GHC.StgToCmm.Expr) jumps here for a tail evaluation,
 + * which getCallMethod reaches only for values whose type rules out functions.
 + *
 + * Three shapes evaluate such a value. Measured on nofib imaginary+spectral
 + * (74 benchmarks, cachegrind instruction counts, both sides built with
 + * -fproc-alignment=64 so that code placement is held fixed):
 + *
 + *   - Enter unconditionally, reading the entry code out of the closure. This
 + *     reaches the entry code of an evaluated value, which the enter-taggable
 + *     invariant rules out. It also costs a load of the info pointer feeding
 + *     an indirect jump, plus the entry code's own indirect jump to the
 + *     continuation, whenever the value is already evaluated.
 + *   - Jump to stg_ap_0_fast. Adds a load of the closure type and a
 + *     bounds-checked switch on it, both dead on this domain: +0.19%
 + *     instructions over this stub.
 + *   - Jump to this stub. +0.07% instructions over entering unconditionally,
 + *     allocations unchanged. The tag test mostly fails, because values whose
 + *     tag is statically known are routed to InferedReturnIt by tag inference
 + *     before they reach here, leaving mostly genuine thunks: fft2 +0.36%,
 + *     fft +0.24%.
 + *
 + * Cycles move in both directions and by more than the instruction counts,
 + * because what the test replaces is a load feeding an indirect jump rather
 + * than straight-line work. Where the value is usually evaluated, primetest
 + * runs -1.5% and exact-reals -1.0%; where it is usually a thunk, fft2 runs
 + * +0.9% and fft +0.4%.
 + *
 + * The stub keeps the test out of the code generator's per-site output. Open
 + * coding it at each call site costs a temporary, two basic blocks and an
 + * explicit return there, which compiling T13960 pays for at around +3%
 + * allocations; through the stub the compiler emits one jump, as it did when
 + * it entered unconditionally.
 + */
++
 +stg_enter_data ( P_ x )
 +{
 +    if (GETTAG(x) != 0) {
 +        return (x);
 +    }
 +    jump %GET_ENTRY(x) (x);
 +}
++
  /* ----------------------------------------------------------------------------
   * Evaluate a closure and return it.
+  *

rts/RtsSymbols.c

@@ -842,6 +842,7 @@ extern char **environ;
        SymI_HasDataProto(stg_ap_ppppp_info)                                  \
        SymI_HasDataProto(stg_ap_pppppp_info)                                 \
        SymI_HasDataProto(stg_ap_0_fast)                                      \
 +      SymI_HasDataProto(stg_enter_data)                                     \
        SymI_HasDataProto(stg_ap_v_fast)                                      \
        SymI_HasDataProto(stg_ap_f_fast)                                      \
        SymI_HasDataProto(stg_ap_d_fast)                                      \

rts/include/stg/MiscClosures.h

@@ -300,6 +300,7 @@ RTS_RET(stg_ap_ppppp);
  RTS_RET(stg_ap_pppppp);
  RTS_FUN_DECL(stg_ap_0_fast);
 +RTS_FUN_DECL(stg_enter_data);
  RTS_FUN_DECL(stg_ap_v_fast);
  RTS_FUN_DECL(stg_ap_f_fast);
  RTS_FUN_DECL(stg_ap_d_fast);