Alan Zimmerman pushed to branch wip/az/ghc-cpp at Glasgow Haskell Compiler / GHC Commits: d47bf776 by Matthew Pickering at 2025-04-14T16:44:41+01:00 driver: Use ModuleGraph for oneshot and --make mode This patch uses the `hsc_mod_graph` field for both oneshot and --make mode. Therefore, if part of the compiler requires usage of the module graph, you do so in a uniform way for the two different modes. The `ModuleGraph` describes the relationship between the modules in the home package and units in external packages. The `ModuleGraph` can be queried when information about the transitive closure of a package is needed. For example, the primary use of the ModuleGraph from within the compiler is in the loader, which needs to know the transitive closure of a module so it can load all the relevant objects for evaluation. In --make mode, downsweep computes the ModuleGraph before any compilation starts. In oneshot mode, a thunk is created at the start of compilation, which when forced will compute the module graph beneath the current module. The thunk is only forced at the moment when the user uses Template Haskell. Finally, there are some situations where we need to discover what dependencies to load but haven't loaded a module graph at all. In this case, there is a fallback which computes the transitive closure on the fly and doesn't cache the result. Presumably if you are going to call getLinkDeps a lot, you would compute the right ModuleGraph before you started. Importantly, this removes the ExternalModuleGraph abstraction. This was quite awkward to work with since it stored information about the home package inside the EPS. This patch will also be very useful when implementing explicit level imports, which requires more significant use of the module graph in order to determine which level instances are available at. Towards #25795 ------------------------- Metric Decrease: MultiLayerModulesTH_Make MultiLayerModulesTH_OneShot ------------------------- - - - - - 395e0ad1 by sheaf at 2025-04-16T12:33:26-04:00 base: remove .Internal modules (e.g. GHC.TypeLits) This commit removes the following internal modules from base, as per CLC proposal 217: - GHC.TypeNats.Internal - GHC.TypeLits.Internal - GHC.ExecutionStack.Internal Fixes #25007 - - - - - 123063e9 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 GHC-CPP: first rough proof of concept Processes #define FOO #ifdef FOO x = 1 #endif Into [ITcppIgnored [L loc ITcppDefine] ,ITcppIgnored [L loc ITcppIfdef] ,ITvarid "x" ,ITequal ,ITinteger (IL {il_text = SourceText "1", il_neg = False, il_value = 1}) ,ITcppIgnored [L loc ITcppEndif] ,ITeof] In time, ITcppIgnored will be pushed into a comment - - - - - fd78ccfb by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Tidy up before re-visiting the continuation mechanic - - - - - 52d3b28a by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Switch preprocessor to continuation passing style Proof of concept, needs tidying up - - - - - d13cf4cb by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Small cleanup - - - - - 824da70d by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Get rid of some cruft - - - - - eb4129f5 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Starting to integrate. Need to get the pragma recognised and set - - - - - aad083ff by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Make cppTokens extend to end of line, and process CPP comments - - - - - 45924ee6 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Remove unused ITcppDefined - - - - - 44e4848c by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Allow spaces between # and keyword for preprocessor directive - - - - - 79969238 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Process CPP continuation lines They are emited as separate ITcppContinue tokens. Perhaps the processing should be more like a comment, and keep on going to the end. BUT, the last line needs to be slurped as a whole. - - - - - f0c652a4 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Accumulate CPP continuations, process when ready Can be simplified further, we only need one CPP token - - - - - f31fe568 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Simplify Lexer interface. Only ITcpp We transfer directive lines through it, then parse them from scratch in the preprocessor. - - - - - bd6770e9 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Deal with directive on last line, with no trailing \n - - - - - f1e92507 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Start parsing and processing the directives - - - - - 57711e91 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Prepare for processing include files - - - - - 9bfc7265 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Move PpState into PreProcess And initParserState, initPragState too - - - - - 7cc6b013 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Process nested include files Also move PpState out of Lexer.x, so it is easy to evolve it in a ghci session, loading utils/check-cpp/Main.hs - - - - - 4b52bccf by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Split into separate files - - - - - 96e31900 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Starting on expression parser. But it hangs. Time for Text.Parsec.Expr - - - - - e49afbce by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Start integrating the ghc-cpp work From https://github.com/alanz/ghc-cpp - - - - - e2cc26ad by Alan Zimmerman at 2025-04-16T20:10:56+01:00 WIP - - - - - 160c9536 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Fixup after rebase - - - - - 88624cf4 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 WIP - - - - - f5926471 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Fixup after rebase, including all tests pass - - - - - a1367108 by Alan Zimmerman at 2025-04-16T20:10:56+01:00 Change pragma usage to GHC_CPP from GhcCPP - - - - - 7e6498c1 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Some comments - - - - - e78ec5c9 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Reformat - - - - - 9944ddb2 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Delete unused file - - - - - d6c5e468 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Rename module Parse to ParsePP - - - - - 90325d92 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Clarify naming in the parser - - - - - d3af5ff1 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 WIP. Switching to alex/happy to be able to work in-tree Since Parsec is not available - - - - - 8da3fda7 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Layering is now correct - GHC lexer, emits CPP tokens - accumulated in Preprocessor state - Lexed by CPP lexer, CPP command extracted, tokens concated with spaces (to get rid of token pasting via comments) - if directive lexed and parsed by CPP lexer/parser, and evaluated - - - - - a330a9f6 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 First example working Loading Example1.hs into ghci, getting the right results ``` {-# LANGUAGE GHC_CPP #-} module Example1 where y = 3 x = "hello" "bye now" foo = putStrLn x ``` - - - - - e5719609 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Rebase, and all tests pass except whitespace for generated parser - - - - - 39bc1059 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 More plumbing. Ready for testing tomorrow. - - - - - 8e0e5340 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Proress. Renamed module State from Types And at first blush it seems to handle preprocessor scopes properly. - - - - - 0e43c398 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Insert basic GHC version macros into parser __GLASGOW_HASKELL__ __GLASGOW_HASKELL_FULL_VERSION__ __GLASGOW_HASKELL_PATCHLEVEL1__ __GLASGOW_HASKELL_PATCHLEVEL2__ - - - - - a9a9eaf0 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Re-sync check-cpp for easy ghci work - - - - - 47dd783d by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Get rid of warnings - - - - - 17bbdb4c by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Rework macro processing, in check-cpp Macros kept at the top level, looked up via name, multiple arity versions per name can be stored - - - - - c440cfe6 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 WIP. Can crack arguments for #define Next step it to crack out args in an expansion - - - - - 8667821a by Alan Zimmerman at 2025-04-16T20:10:57+01:00 WIP on arg parsing. - - - - - 57b11c08 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Progress. Still screwing up nested parens. - - - - - d653f23c by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Seems to work, but has redundant code - - - - - e538ed75 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Remove redundant code - - - - - 0b6d5076 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Reformat - - - - - fff97636 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Expand args, single pass Still need to repeat until fixpoint - - - - - 793a5a50 by Alan Zimmerman at 2025-04-16T20:10:57+01:00 Fixed point expansion - - - - - 9960bc20 by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Sync the playground to compiler - - - - - cf1ba1c3 by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Working on dumping the GHC_CPP result But We need to keep the BufSpan in a comment - - - - - 607a90d1 by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Keep BufSpan in queued comments in GHC.Parser.Lexer - - - - - bb62e04e by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Getting close to being able to print the combined tokens showing what is in and what is out - - - - - 5a94b451 by Alan Zimmerman at 2025-04-16T20:10:58+01:00 First implementation of dumpGhcCpp. Example output First dumps all macros in the state, then the source, showing which lines are in and which are out ------------------------------ - |#define FOO(A,B) A + B - |#define FOO(A,B,C) A + B + C - |#if FOO(1,FOO(3,4)) == 8 - |-- a comment |x = 1 - |#else - |x = 5 - |#endif - - - - - 293695de by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Clean up a bit - - - - - f914e0ee by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Add -ddump-ghc-cpp option and a test based on it - - - - - 5f8f7fce by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Restore Lexer.x rules, we need them for continuation lines - - - - - cdfd5ae3 by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Lexer.x: trying to sort out the span for continuations - We need to match on \n at the end of the line - We cannot simply back up for it - - - - - fa3c0c39 by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Inserts predefined macros. But does not dump properly Because the cpp tokens have a trailing newline - - - - - 43bc181e by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Remove unnecessary LExer rules We *need* the ones that explicitly match to the end of the line. - - - - - 677fbf16 by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Generate correct span for ITcpp Dump now works, except we do not render trailing `\` for continuation lines. This is good enough for use in test output. - - - - - e4340715 by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Reduce duplication in lexer - - - - - 71d4bbf1 by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Tweaks - - - - - b202952e by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Insert min_version predefined macros into state The mechanism now works. Still need to flesh out the full set. - - - - - 40a69c9d by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Trying my alternative pragma syntax. It works, but dumpGhcCpp is broken, I suspect from the ITcpp token span update. - - - - - 4a47227d by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Pragma extraction now works, with both CPP and GHC_CPP For the following {-# LANGUAGE CPP #-} #if __GLASGOW_HASKELL__ >= 913 {-# LANGUAGE GHC_CPP #-} #endif We will enable GHC_CPP only - - - - - 0acda1b0 by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Remove some tracing - - - - - 9110a0ee by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Fix test exes for changes - - - - - 3e1238ed by Alan Zimmerman at 2025-04-16T20:10:58+01:00 For GHC_CPP tests, normalise config-time-based macros - - - - - a39358b9 by Alan Zimmerman at 2025-04-16T20:10:58+01:00 WIP - - - - - b1a3506d by Alan Zimmerman at 2025-04-16T20:10:58+01:00 WIP again. What is wrong? - - - - - 404ca306 by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Revert to dynflags for normal not pragma lexing - - - - - 2c02292a by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Working on getting check-exact to work properly - - - - - a7b57ca5 by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Passes CppCommentPlacement test - - - - - a0064092 by Alan Zimmerman at 2025-04-16T20:10:58+01:00 Starting on exact printing with GHC_CPP While overriding normal CPP - - - - - f9fc3f3e by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Correctly store CPP ignored tokens as comments By populating the lexeme string in it, based on the bufpos - - - - - 37ed0ade by Alan Zimmerman at 2025-04-16T20:10:59+01:00 WIP - - - - - 005347ce by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Simplifying - - - - - 218df0c2 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Update the active state logic - - - - - 03f4dc01 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Work the new logic into the mainline code - - - - - 5dbe86c3 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Process `defined` operator - - - - - 9597084f by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Manage lexer state while skipping tokens There is very intricate layout-related state used when lexing. If a CPP directive blanks out some tokens, store this state when the blanking starts, and restore it when they are no longer being blanked. - - - - - 5bc84360 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Track the last token buffer index, for ITCppIgnored We need to attach the source being skipped in an ITCppIgnored token. We cannot simply use its BufSpan as an index into the underlying StringBuffer as it counts unicode chars, not bytes. So we update the lexer state to store the starting StringBuffer location for the last token, and use the already-stored length to extract the correct portion of the StringBuffer being parsed. - - - - - cfcb6cbf by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Process the ! operator in GHC_CPP expressions - - - - - 81c74e50 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Predefine a constant when GHC_CPP is being used. - - - - - c12468a2 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 WIP - - - - - ba57f7c9 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Skip lines directly in the lexer when required - - - - - df7d3e65 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Properly manage location when accepting tokens again - - - - - 22027a8f by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Seems to be working now, for Example9 - - - - - aded1557 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Remove tracing - - - - - 1ce351d1 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Fix parsing '*' in block comments Instead of replacing them with '-' - - - - - c8e3ce16 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Keep the trailing backslash in a ITcpp token - - - - - ed91afa3 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Deal with only enabling one section of a group. A group is an instance of a conditional introduced by #if/#ifdef/#ifndef, and ending at the final #endif, including intermediate #elsif sections - - - - - 76355312 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Replace remaining identifiers with 0 when evaluating As per the spec - - - - - 0cccd058 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Snapshot before rebase - - - - - 899c3a8c by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Skip non-processed lines starting with # - - - - - 1b186f6c by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Export generateMacros so we can use it in ghc-exactprint - - - - - d3e2e345 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Fix rebase - - - - - 37a65b77 by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Expose initParserStateWithMacrosString - - - - - 20ffbaba by Alan Zimmerman at 2025-04-16T20:10:59+01:00 Fix buggy lexer cppSkip It was skipping all lines, not just ones prefixed by # - - - - - 39aae9d4 by Alan Zimmerman at 2025-04-16T20:11:00+01:00 Fix evaluation of && to use the correct operator - - - - - 608bf45b by Alan Zimmerman at 2025-04-16T20:11:00+01:00 Deal with closing #-} at the start of a line - - - - - eba2d8f4 by Alan Zimmerman at 2025-04-16T20:11:00+01:00 Add the MIN_VERSION_GLASGOW_HASKELL predefined macro - - - - - cf231960 by Alan Zimmerman at 2025-04-16T20:11:00+01:00 Include MIN_VERSION_GLASGOW_HASKELL in GhcCpp01.stderr - - - - - a92c63d3 by Alan Zimmerman at 2025-04-16T20:11:00+01:00 Use a strict map for macro defines - - - - - 109 changed files: - compiler/GHC.hs - compiler/GHC/Cmm/Lexer.x - compiler/GHC/Cmm/Parser.y - compiler/GHC/Cmm/Parser/Monad.hs - compiler/GHC/Driver/Backpack.hs - compiler/GHC/Driver/Config/Parser.hs - compiler/GHC/Driver/Downsweep.hs - compiler/GHC/Driver/Env.hs - compiler/GHC/Driver/Env/Types.hs - compiler/GHC/Driver/Errors/Ppr.hs - compiler/GHC/Driver/Errors/Types.hs - compiler/GHC/Driver/Flags.hs - compiler/GHC/Driver/Main.hs - compiler/GHC/Driver/Make.hs - compiler/GHC/Driver/MakeAction.hs - + compiler/GHC/Driver/Messager.hs - compiler/GHC/Driver/Pipeline.hs - compiler/GHC/Driver/Pipeline.hs-boot - compiler/GHC/Driver/Pipeline/Execute.hs - compiler/GHC/Driver/Session.hs - compiler/GHC/Iface/Load.hs - compiler/GHC/Iface/Recomp.hs - compiler/GHC/Linker/Deps.hs - compiler/GHC/Linker/Loader.hs - compiler/GHC/Parser.hs-boot - compiler/GHC/Parser.y - compiler/GHC/Parser/Annotation.hs - compiler/GHC/Parser/HaddockLex.x - compiler/GHC/Parser/Header.hs - compiler/GHC/Parser/Lexer.x - compiler/GHC/Parser/PostProcess.hs - compiler/GHC/Parser/PostProcess/Haddock.hs - + compiler/GHC/Parser/PreProcess.hs - + compiler/GHC/Parser/PreProcess/Eval.hs - + compiler/GHC/Parser/PreProcess/Lexer.x - + compiler/GHC/Parser/PreProcess/Macro.hs - + compiler/GHC/Parser/PreProcess/ParsePP.hs - + compiler/GHC/Parser/PreProcess/Parser.y - + compiler/GHC/Parser/PreProcess/ParserM.hs - + compiler/GHC/Parser/PreProcess/State.hs - compiler/GHC/Parser/Utils.hs - compiler/GHC/Runtime/Loader.hs - compiler/GHC/SysTools/Cpp.hs - compiler/GHC/Unit/External.hs - compiler/GHC/Unit/Finder.hs - − compiler/GHC/Unit/Module/External/Graph.hs - compiler/GHC/Unit/Module/Graph.hs - compiler/GHC/Unit/Module/ModNodeKey.hs - compiler/ghc.cabal.in - docs/users_guide/debugging.rst - ghc/GHCi/UI.hs - hadrian/src/Rules/SourceDist.hs - hadrian/stack.yaml.lock - libraries/base/base.cabal.in - libraries/base/changelog.md - − libraries/base/src/GHC/ExecutionStack/Internal.hs - − libraries/base/src/GHC/TypeLits/Internal.hs - − libraries/base/src/GHC/TypeNats/Internal.hs - libraries/ghc-internal/src/GHC/Internal/LanguageExtensions.hs - testsuite/tests/count-deps/CountDepsParser.stdout - testsuite/tests/driver/T4437.hs - testsuite/tests/ghc-api/T11579.hs - + testsuite/tests/ghc-api/fixed-nodes/InterfaceModuleGraph.hs - + testsuite/tests/ghc-api/fixed-nodes/InterfaceModuleGraph.stdout - testsuite/tests/ghc-api/fixed-nodes/all.T - + testsuite/tests/ghc-cpp/GhcCpp01.hs - + testsuite/tests/ghc-cpp/GhcCpp01.stderr - + testsuite/tests/ghc-cpp/all.T - testsuite/tests/interface-stability/base-exports.stdout - testsuite/tests/interface-stability/base-exports.stdout-javascript-unknown-ghcjs - testsuite/tests/interface-stability/base-exports.stdout-mingw32 - testsuite/tests/interface-stability/base-exports.stdout-ws-32 - testsuite/tests/interface-stability/template-haskell-exports.stdout - testsuite/tests/plugins/defaulting-plugin/DefaultLifted.hs - + testsuite/tests/printer/CppCommentPlacement.hs - + utils/check-cpp/.ghci - + utils/check-cpp/.gitignore - + utils/check-cpp/Eval.hs - + utils/check-cpp/Example1.hs - + utils/check-cpp/Example10.hs - + utils/check-cpp/Example11.hs - + utils/check-cpp/Example12.hs - + utils/check-cpp/Example13.hs - + utils/check-cpp/Example2.hs - + utils/check-cpp/Example3.hs - + utils/check-cpp/Example4.hs - + utils/check-cpp/Example5.hs - + utils/check-cpp/Example6.hs - + utils/check-cpp/Example7.hs - + utils/check-cpp/Example8.hs - + utils/check-cpp/Example9.hs - + utils/check-cpp/Lexer.x - + utils/check-cpp/Macro.hs - + utils/check-cpp/Main.hs - + utils/check-cpp/ParsePP.hs - + utils/check-cpp/ParseSimulate.hs - + utils/check-cpp/Parser.y - + utils/check-cpp/ParserM.hs - + utils/check-cpp/PreProcess.hs - + utils/check-cpp/README.md - + utils/check-cpp/State.hs - + utils/check-cpp/run.sh - utils/check-exact/Main.hs - utils/check-exact/Parsers.hs - utils/check-exact/Preprocess.hs - utils/check-exact/Utils.hs - utils/haddock/haddock-api/src/Haddock/Backends/Hyperlinker/Parser.hs - utils/haddock/haddock-api/src/Haddock/Parser.hs - utils/haddock/haddock-api/src/Haddock/Types.hs The diff was not included because it is too large. View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/6ff34ae924b99fa7abe1f96b1741924... -- View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/6ff34ae924b99fa7abe1f96b1741924... You're receiving this email because of your account on gitlab.haskell.org.