Alan Zimmerman pushed to branch wip/az/ghc-cpp at Glasgow Haskell Compiler / GHC Commits: 14281a22 by Ben Gamari at 2025-10-11T14:06:47-04:00 rts/nonmoving: Fix comment spelling - - - - - bedd38b0 by Ben Gamari at 2025-10-11T14:06:47-04:00 rts/nonmoving: Use atomic operations to update bd->flags - - - - - 215d6841 by Ben Gamari at 2025-10-11T14:06:47-04:00 nonmoving: Use get_itbl instead of explicit loads This is cleaner and also fixes unnecessary (and unsound) use of `volatile`. - - - - - 2c94aa3a by Ben Gamari at 2025-10-11T14:06:47-04:00 rts/Scav: Handle WHITEHOLEs in scavenge_one `scavenge_one`, used to scavenge mutable list entries, may encounter `WHITEHOLE`s when the non-moving GC is in use via two paths: 1. when an MVAR is being marked concurrently 2. when the object belongs to a chain of selectors being short-cutted. Fixes #26204. - - - - - 6bd8155c by Matthew Pickering at 2025-10-11T14:07:29-04:00 Add support for generating bytecode objects This commit adds the `-fwrite-byte-code` option which makes GHC emit a `.gbc` file which contains a serialised representation of bytecode. The bytecode can be loaded by the compiler to avoid having to reinterpret a module when using the bytecode interpreter (for example, in GHCi). There are also the new options: * -gbcdir=<DIR>: Specify the directory to place the gbc files * -gbcsuf=<suffix>: Specify the suffix for gbc files The option `-fbyte-code-and-object-code` now implies `-fwrite-byte-code`. These performance tests fail due to https://github.com/haskell/directory/issues/204 ------------------------- Metric Increase: MultiComponentModules MultiLayerModules MultiComponentModulesRecomp MultiLayerModulesRecomp MultiLayerModulesTH_Make MultiLayerModulesTH_OneShot T13701 ------------------------- The bytecode serialisation part was implemented by Cheng Shao Co-authored-by: Cheng Shao <terrorjack@type.dance> - - - - - dc8f9599 by Matthew Pickering at 2025-10-11T14:07:30-04:00 Revert "Add a perf test for #26425" This test has a large memory spike currently, which makes the test sensitive, since if you allocate a little more or less, the precise location where GC happens shifts and you observe a different part of the spike. Andreas told me to revert the patch for now, and he will add it back when he fixes the memory spike. This reverts commit 41bdb16fd083110a06507248f648c507a2feb4af. - - - - - e10dcd65 by Sven Tennie at 2025-10-12T10:24:56+00:00 T22859: Increase threadDelay for small machines The previously used thread delay led to failures on my RISC-V test setups. - - - - - d59ef6b6 by Hai / @BestYeen at 2025-10-14T21:51:14-04:00 Change Alex and Happy m4 scripts to display which version was found in the system, adapt small formatting details in Happy script to be more like the Alex script again. - - - - - c98abb6a by Hai / @BestYeen at 2025-10-14T21:52:08-04:00 Update occurrences of return to pure and add a sample for redefining :m to mean :main - - - - - 70ee825a by Cheng Shao at 2025-10-14T21:52:50-04:00 testsuite: fix T3586 for non-SSE3 platforms `T3586.hs` contains `-fvia-C -optc-msse3` which I think is a best-effort basis to harvest the C compiler's auto vectorization optimizations via the C backend back when the test was added. The `-fvia-C` part is now a deprecated no-op because GHC can't fall back to the C backend on a non-unregisterised build, and `-optc-msse3` might actually cause the test to fail on non x86/x64 platforms, e.g. recent builds of wasi-sdk would report `wasm32-wasi-clang: error: unsupported option '-msse3' for target 'wasm32-unknown-wasi'`. So this patch cleans up this historical cruft. `-fvia-C` is removed, and `-optc-msse3` is only passed when cpuid contains `pni` (which indicates support of SSE3). - - - - - 4be32153 by Teo Camarasu at 2025-10-15T08:06:09-04:00 Add submodules for template-haskell-lift and template-haskell-quasiquoter These two new boot libraries expose stable subsets of the template-haskell interface. This is an implemenation of the GHC proposal https://github.com/ghc-proposals/ghc-proposals/pull/696 Work towards #25262 - - - - - 0c00c9c3 by Ben Gamari at 2025-10-15T08:06:51-04:00 rts: Eliminate uses of implicit constant arrays Folding of `const`-sized variable-length arrays to a constant-length array is a gnu extension which clang complains about. Closes #26502. - - - - - 670e8dd1 by Alan Zimmerman at 2025-10-15T20:27:21+01:00 GHC-CPP: Initial implementation Processes #define FOO #ifdef FOO x = 1 #endif Into [ITcppIgnored [L loc ITcppDefine] ,ITcppIgnored [L loc ITcppIfdef] ,ITvarid "x" ,ITequal ,ITinteger (IL {il_text = SourceText "1", il_neg = False, il_value = 1}) ,ITcppIgnored [L loc ITcppEndif] ,ITeof] In time, ITcppIgnored will be pushed into a comment Tidy up before re-visiting the continuation mechanic Switch preprocessor to continuation passing style Proof of concept, needs tidying up Small cleanup Get rid of some cruft Summary: Patch: Author: Alan Zimmerman <alan.zimm@gmail.com> Date: 2025-10-12 16:23:56 +0100 Summary: Patch: rebase-and-tests-pass Author: Alan Zimmerman <alan.zimm@gmail.com> Date: 2025-10-12 14:19:04 +0100 Rebase, and all tests pass except whitespace for generated parser Starting to integrate. Need to get the pragma recognised and set Make cppTokens extend to end of line, and process CPP comments Remove unused ITcppDefined Allow spaces between # and keyword for preprocessor directive Process CPP continuation lines They are emited as separate ITcppContinue tokens. Perhaps the processing should be more like a comment, and keep on going to the end. BUT, the last line needs to be slurped as a whole. Accumulate CPP continuations, process when ready Can be simplified further, we only need one CPP token Simplify Lexer interface. Only ITcpp We transfer directive lines through it, then parse them from scratch in the preprocessor. Deal with directive on last line, with no trailing \n Start parsing and processing the directives Prepare for processing include files Move PpState into PreProcess And initParserState, initPragState too Process nested include files Also move PpState out of Lexer.x, so it is easy to evolve it in a ghci session, loading utils/check-cpp/Main.hs Split into separate files Starting on expression parser. But it hangs. Time for Text.Parsec.Expr Start integrating the ghc-cpp work From https://github.com/alanz/ghc-cpp WIP Fixup after rebase WIP Fixup after rebase, including all tests pass Change pragma usage to GHC_CPP from GhcCPP Some comments Reformat Delete unused file Rename module Parse to ParsePP Clarify naming in the parser WIP. Switching to alex/happy to be able to work in-tree Since Parsec is not available Layering is now correct - GHC lexer, emits CPP tokens - accumulated in Preprocessor state - Lexed by CPP lexer, CPP command extracted, tokens concated with spaces (to get rid of token pasting via comments) - if directive lexed and parsed by CPP lexer/parser, and evaluated First example working Loading Example1.hs into ghci, getting the right results ``` {-# LANGUAGE GHC_CPP #-} module Example1 where y = 3 x = "hello" "bye now" foo = putStrLn x ``` Rebase, and all tests pass except whitespace for generated parser info: patch template saved to `-` More plumbing. Ready for testing tomorrow. Proress. Renamed module State from Types And at first blush it seems to handle preprocessor scopes properly. Insert basic GHC version macros into parser __GLASGOW_HASKELL__ __GLASGOW_HASKELL_FULL_VERSION__ __GLASGOW_HASKELL_PATCHLEVEL1__ __GLASGOW_HASKELL_PATCHLEVEL2__ Re-sync check-cpp for easy ghci work Get rid of warnings Rework macro processing, in check-cpp Macros kept at the top level, looked up via name, multiple arity versions per name can be stored WIP. Can crack arguments for #define Next step it to crack out args in an expansion WIP on arg parsing. Progress. Still screwing up nested parens. Seems to work, but has redundant code Remove redundant code Reformat Expand args, single pass Still need to repeat until fixpoint Fixed point expansion Sync the playground to compiler Working on dumping the GHC_CPP result But We need to keep the BufSpan in a comment Keep BufSpan in queued comments in GHC.Parser.Lexer Getting close to being able to print the combined tokens showing what is in and what is out First implementation of dumpGhcCpp. Example output First dumps all macros in the state, then the source, showing which lines are in and which are out ------------------------------ - |#define FOO(A,B) A + B - |#define FOO(A,B,C) A + B + C - |#if FOO(1,FOO(3,4)) == 8 - |-- a comment |x = 1 - |#else - |x = 5 - |#endif Clean up a bit Add -ddump-ghc-cpp option and a test based on it Restore Lexer.x rules, we need them for continuation lines Lexer.x: trying to sort out the span for continuations - We need to match on \n at the end of the line - We cannot simply back up for it Inserts predefined macros. But does not dump properly Because the cpp tokens have a trailing newline Remove unnecessary LExer rules We *need* the ones that explicitly match to the end of the line. Generate correct span for ITcpp Dump now works, except we do not render trailing `\` for continuation lines. This is good enough for use in test output. Reduce duplication in lexer Tweaks Insert min_version predefined macros into state The mechanism now works. Still need to flesh out the full set. Trying my alternative pragma syntax. It works, but dumpGhcCpp is broken, I suspect from the ITcpp token span update. Pragma extraction now works, with both CPP and GHC_CPP For the following {-# LANGUAGE CPP #-} #if __GLASGOW_HASKELL__ >= 913 {-# LANGUAGE GHC_CPP #-} #endif We will enable GHC_CPP only Remove some tracing Fix test exes for changes For GHC_CPP tests, normalise config-time-based macros WIP WIP again. What is wrong? Revert to dynflags for normal not pragma lexing Working on getting check-exact to work properly Passes CppCommentPlacement test Starting on exact printing with GHC_CPP While overriding normal CPP Correctly store CPP ignored tokens as comments By populating the lexeme string in it, based on the bufpos WIP Simplifying Update the active state logic Work the new logic into the mainline code Process `defined` operator Manage lexer state while skipping tokens There is very intricate layout-related state used when lexing. If a CPP directive blanks out some tokens, store this state when the blanking starts, and restore it when they are no longer being blanked. Track the last token buffer index, for ITCppIgnored We need to attach the source being skipped in an ITCppIgnored token. We cannot simply use its BufSpan as an index into the underlying StringBuffer as it counts unicode chars, not bytes. So we update the lexer state to store the starting StringBuffer location for the last token, and use the already-stored length to extract the correct portion of the StringBuffer being parsed. Process the ! operator in GHC_CPP expressions Predefine a constant when GHC_CPP is being used. WIP Skip lines directly in the lexer when required Properly manage location when accepting tokens again Seems to be working now, for Example9 Remove tracing Fix parsing '*' in block comments Instead of replacing them with '-' Keep the trailing backslash in a ITcpp token Deal with only enabling one section of a group. A group is an instance of a conditional introduced by #if/#ifdef/#ifndef, and ending at the final #endif, including intermediate #elsif sections Replace remaining identifiers with 0 when evaluating As per the spec Snapshot before rebase Skip non-processed lines starting with # Export generateMacros so we can use it in ghc-exactprint Fix rebase Expose initParserStateWithMacrosString Fix buggy lexer cppSkip It was skipping all lines, not just ones prefixed by # Fix evaluation of && to use the correct operator Deal with closing #-} at the start of a line Add the MIN_VERSION_GLASGOW_HASKELL predefined macro Include MIN_VERSION_GLASGOW_HASKELL in GhcCpp01.stderr Use a strict map for macro defines Process TIdentifierLParen Which only matters at the start of #define Do not provide TIdentifierLParen paren twice Handle whitespace between identifier and '(' for directive only Expose some Lexer bitmap manipulation helpers Deal with line pragmas as tokens Blows up for dumpGhcCpp though Allow strings delimited by a single quote too Allow leading whitespace on cpp directives As per https://timsong-cpp.github.io/cppwp/n4140/cpp#1 Implement GHC_CPP undef Sort out expansion of no-arg macros, in a context with args And make the expansion bottom out, in the case of recursion Fix GhcCpp01 test The LINE pragma stuff works in ghc-exactprint when specifically setting flag to emit ITline_pragma tokens Process comments in CPP directives Correctly lex pragmas with finel #-} on a newline Do not process CPP-style comments Allow cpp-style comments when GHC_CPP enabled Return other pragmas as cpp ignored when GHC_CPP active Reorganise getOptionsFromFile for use in ghc-exactprint We want to be able to inject predefined macro definitions into the parser preprocessor state for when we do a hackage roundtrip. Tweak testing Only allow unknown cpp pragmas with # in left margin Require # against left margin for all GHC_CPP directives Fix CPP directives appearing in pragmas And add a test for error reporting for missing `#if` Starting to report GHC_CPP errors using GHC machinery More GHC_CPP diagnostic results WIP on converting error calls to GHC diagnostics in GHC_CPP Working on CPP diagnostic reporting Tweak some tests/lint warnings More error reporting in Macro Some cleanups Some cleanup GHC_CPP: Working on improving error reporting Harvest some commonality Use PPM as Maybe inside PP Clean up a bit Fix GhcCpp01 test I think this needs to be made more robust. Likely by not dumping the (pre-)defined macros. info: patch template saved to `-` info: patch template saved to `-` - - - - - b3280001 by Alan Zimmerman at 2025-10-15T21:25:09+01:00 Fix GhcCpp01 after rebase - - - - - 847eea2c by Alan Zimmerman at 2025-10-15T21:25:09+01:00 GHC_CPP: delete utils/check-cpp It is not needed - - - - - d05c9946 by Alan Zimmerman at 2025-10-15T21:25:09+01:00 GHC_CPP: remove #include processing We do not support it - - - - - 537a881b by Alan Zimmerman at 2025-10-15T21:25:09+01:00 GHC_CPP: move Eval.hs into GHC.Parser.PreProcess.Macro - - - - - 135 changed files: - .gitmodules - compiler/GHC.hs - compiler/GHC/Builtin/PrimOps.hs - compiler/GHC/ByteCode/Breakpoints.hs - + compiler/GHC/ByteCode/Serialize.hs - compiler/GHC/ByteCode/Types.hs - compiler/GHC/Cmm/Lexer.x - compiler/GHC/Cmm/Parser.y - compiler/GHC/Cmm/Parser/Monad.hs - compiler/GHC/Data/FlatBag.hs - compiler/GHC/Data/SmallArray.hs - compiler/GHC/Driver/Backend.hs - compiler/GHC/Driver/Backpack.hs - + compiler/GHC/Driver/ByteCode.hs - compiler/GHC/Driver/Config/Finder.hs - compiler/GHC/Driver/Config/Parser.hs - compiler/GHC/Driver/Downsweep.hs - compiler/GHC/Driver/DynFlags.hs - compiler/GHC/Driver/Flags.hs - compiler/GHC/Driver/Main.hs - compiler/GHC/Driver/Messager.hs - compiler/GHC/Driver/Pipeline.hs - compiler/GHC/Driver/Pipeline/Execute.hs - compiler/GHC/Driver/Session.hs - compiler/GHC/HsToCore/Breakpoints.hs - compiler/GHC/HsToCore/Usage.hs - compiler/GHC/Linker/Deps.hs - compiler/GHC/Linker/Loader.hs - compiler/GHC/Linker/Types.hs - compiler/GHC/Parser.hs-boot - compiler/GHC/Parser.y - compiler/GHC/Parser/Annotation.hs - compiler/GHC/Parser/Errors/Ppr.hs - compiler/GHC/Parser/Errors/Types.hs - compiler/GHC/Parser/HaddockLex.x - compiler/GHC/Parser/Header.hs - compiler/GHC/Parser/Lexer.x - compiler/GHC/Parser/PostProcess.hs - compiler/GHC/Parser/PostProcess/Haddock.hs - + compiler/GHC/Parser/PreProcess.hs - + compiler/GHC/Parser/PreProcess/Lexer.x - + compiler/GHC/Parser/PreProcess/Macro.hs - + compiler/GHC/Parser/PreProcess/ParsePP.hs - + compiler/GHC/Parser/PreProcess/Parser.y - + compiler/GHC/Parser/PreProcess/ParserM.hs - + compiler/GHC/Parser/PreProcess/State.hs - compiler/GHC/Parser/Utils.hs - compiler/GHC/Runtime/Eval.hs - compiler/GHC/SysTools/Cpp.hs - compiler/GHC/Types/Error/Codes.hs - compiler/GHC/Types/SptEntry.hs - compiler/GHC/Types/Tickish.hs - compiler/GHC/Unit/Finder.hs - compiler/GHC/Unit/Finder/Types.hs - compiler/GHC/Unit/Module/Graph.hs - compiler/GHC/Unit/Module/Location.hs - compiler/GHC/Unit/Module/ModSummary.hs - compiler/GHC/Unit/Module/WholeCoreBindings.hs - compiler/GHC/Utils/Binary.hs - compiler/ghc.cabal.in - docs/users_guide/debugging.rst - docs/users_guide/ghci.rst - docs/users_guide/phases.rst - docs/users_guide/separate_compilation.rst - ghc/GHCi/UI.hs - hadrian/src/Packages.hs - hadrian/src/Rules/SourceDist.hs - hadrian/src/Settings/Default.hs - libraries/ghc-internal/src/GHC/Internal/LanguageExtensions.hs - libraries/ghc-internal/src/GHC/Internal/TH/Lift.hs - libraries/ghc-internal/src/GHC/Internal/TH/Quote.hs - + libraries/template-haskell-lift - + libraries/template-haskell-quasiquoter - m4/fptools_alex.m4 - m4/fptools_happy.m4 - rts/Printer.c - rts/include/rts/storage/Block.h - rts/posix/OSMem.c - rts/sm/NonMoving.c - rts/sm/NonMovingMark.c - rts/sm/Scav.c - testsuite/driver/cpu_features.py - testsuite/tests/bytecode/T24634/T24634a.stdout - testsuite/tests/bytecode/T24634/T24634b.stdout - testsuite/tests/count-deps/CountDepsAst.stdout - testsuite/tests/count-deps/CountDepsParser.stdout - testsuite/tests/driver/T4437.hs - + testsuite/tests/driver/bytecode-object/A.hs - + testsuite/tests/driver/bytecode-object/BytecodeForeign.c - + testsuite/tests/driver/bytecode-object/BytecodeForeign.hs - + testsuite/tests/driver/bytecode-object/BytecodeMain.hs - + testsuite/tests/driver/bytecode-object/BytecodeTest.hs - + testsuite/tests/driver/bytecode-object/Makefile - + testsuite/tests/driver/bytecode-object/all.T - + testsuite/tests/driver/bytecode-object/bytecode_object12.stderr - + testsuite/tests/driver/bytecode-object/bytecode_object13.stdout - + testsuite/tests/driver/bytecode-object/bytecode_object14.stdout - + testsuite/tests/driver/bytecode-object/bytecode_object15.stdout - + testsuite/tests/driver/bytecode-object/bytecode_object16.stdout - + testsuite/tests/driver/bytecode-object/bytecode_object17.stdout - + testsuite/tests/driver/bytecode-object/bytecode_object18.stdout - + testsuite/tests/driver/bytecode-object/bytecode_object19.script - + testsuite/tests/driver/bytecode-object/bytecode_object19.stdout - + testsuite/tests/driver/bytecode-object/bytecode_object25.stdout - + testsuite/tests/driver/bytecode-object/bytecode_object4.stdout - + testsuite/tests/driver/bytecode-object/bytecode_object5.stdout - + testsuite/tests/driver/bytecode-object/bytecode_object6.stdout - testsuite/tests/driver/fat-iface/T22405/T22405.stdout - testsuite/tests/driver/fat-iface/T22405/T22405b.stdout - testsuite/tests/driver/fat-iface/fat011.stderr - testsuite/tests/ghc-api/T11579.hs - + testsuite/tests/ghc-cpp/GhcCpp01.hs - + testsuite/tests/ghc-cpp/GhcCpp01.stderr - + testsuite/tests/ghc-cpp/GhcCpp02.hs - + testsuite/tests/ghc-cpp/GhcCpp02.stderr - + testsuite/tests/ghc-cpp/all.T - testsuite/tests/interface-stability/template-haskell-exports.stdout - testsuite/tests/linters/notes.stdout - testsuite/tests/linters/regex-linters/check-cpp.py - testsuite/tests/perf/compiler/Makefile - + testsuite/tests/perf/compiler/MultiLayerModulesDefsGhciWithBytecodeFiles.script - − testsuite/tests/perf/compiler/T26425.hs - testsuite/tests/perf/compiler/all.T - testsuite/tests/perf/should_run/T3586.hs - testsuite/tests/perf/should_run/all.T - + testsuite/tests/printer/CppCommentPlacement.hs - testsuite/tests/rts/T22859.hs - testsuite/tests/simplStg/should_compile/T22840.stderr - utils/check-exact/Main.hs - utils/check-exact/Parsers.hs - utils/check-exact/Preprocess.hs - utils/check-exact/Utils.hs - utils/haddock/haddock-api/src/Haddock/Backends/Hyperlinker/Parser.hs - utils/haddock/haddock-api/src/Haddock/Parser.hs - utils/haddock/haddock-api/src/Haddock/Types.hs The diff was not included because it is too large. View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/ac6bbca6d8dc44f5e725a3daa85a60c... -- View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/ac6bbca6d8dc44f5e725a3daa85a60c... You're receiving this email because of your account on gitlab.haskell.org.