Sven Tennie pushed to branch wip/supersven/hadrian-cross-stage3 at Glasgow Haskell Compiler / GHC Commits: de6bc053 by Sven Tennie at 2026-10-10T12:04:04+00:00 hadrian: Add Stage3 cross-compiled target bindist (#26924) Add a `binary-dist-stage3` Hadrian target that packages target executables (produced by the stage2 compiler) into a separate '_build/bindist-stage3/' folder, distinct from the stage2 regular or cross-compiler bindist in '_build/bindist/'. This allows a single CI pipeline job to produce both a cross-compiler bindist (e.g. x86_64 -> RISC-V) and a target-architecture bindist (e.g. RISC-V -> RISC-V) that can be installed and run natively on the target. To avoid issues with stale files or race-conditions on them, generate the `configure` script per stage in `_build/<stage>/distrib` directories. - - - - - be47afa7 by Sven Tennie at 2026-10-10T12:04:04+00:00 ci: Build Stage3 and Stage2 bindists for cross targets in one job (#26924) Building the stage3 target bindist already produces most of the stage2 cross-compiler bindist as a byproduct, so building them in separate jobs duplicates the build efforts for no benefit. Instead of a separate CROSS_STAGE=3 job, the stage3 job now also builds and publishes the stage2 bindist. The stage3 tarball is named and versioned as if it were built natively on the target (target triple prefix, non-cross opsys), so its artifact name stays what downstream consumers already expect and no changes are needed on their side. For now, this is only enabled for the RISC-V job. Others can easily follow. - - - - - 62724e75 by Sven Tennie at 2026-10-10T12:04:04+00:00 ci: model cross targets as structured TargetPlatform triples Storing the cross target as a raw triple forced ad-hoc parsing at every use, and broke the single responsibility rule. Modelling it as a structured `TargetPlatform`` instead, avoids parsing and provides a clear separation of concerns. - - - - - f0446a13 by Sven Tennie at 2026-10-10T12:04:04+00:00 ci: use a scalar cross-stage switch, FINAL_CROSS_STAGE=2|3 For cross-compilers, Stage2 bindists are (almost) a by-product of Stage3 bindists. Thus, we either build Stage2 or Stage2 + Stage3 bindists. The commit at hand reflects this fact in types. - - - - - 72b47772 by Sven Tennie at 2026-10-10T12:04:04+00:00 ci: extract the cross smoke tests into a separate smoke_test command This reduces if-nesting and complexity of the main `test_hadrian` command. The decision how to test is now made by the caller. This is a fair choice, because most test cases can run under emulation - they are just very slow then. - - - - - 8997872c by Sven Tennie at 2026-10-10T12:04:04+00:00 ci: inline the FINAL_CROSS_STAGE renderer `finalCrossStageEnvVal` was small and only used in one place. - - - - - d096e3c6 by Sven Tennie at 2026-10-10T12:04:04+00:00 hadrian: clarify cross-stage settings and project.mk comments Reword the settings-rule comment to name the two stage roles (compilerStage vs libraryStage) and explain what 'the final architecture' in terms of project.mk generation means. - - - - - 9 changed files: - .gitignore - .gitlab/ci.sh - .gitlab/generate-ci/gen_ci.hs - .gitlab/jobs.yaml - + changelog.d/stage3-cross-bindists - distrib/configure.ac.in - hadrian/src/BindistConfig.hs - hadrian/src/Rules/BinaryDist.hs - hadrian/src/Rules/Generate.hs Changes: ===================================== .gitignore ===================================== @@ -120,7 +120,6 @@ _darcs/ /compiler/GHC/CmmToLlvm/Version/Bounds.hs /compiler/ghc.cabal /compiler/ghc.cabal.old -/distrib/configure.ac /distrib/ghc.iss /docs/index.html /docs/man ===================================== .gitlab/ci.sh ===================================== @@ -62,9 +62,10 @@ Common Modes: Environment variables affecting the build: CROSS_TARGET Triple of cross-compilation target. - CROSS_STAGE The stage of the cross-compiler to build either - * 2: Build a normal cross-compiler bindist - * 3: Build a target executable bindist (with the stage2 cross-compiler) + FINAL_CROSS_STAGE The final stage of the cross-compiler to build either + * 2: Build a cross-compiler bindist + * 3: Also build the target executable bindist + (builds stages 2 and 3; implies the bindist of 2) VERBOSE Set to non-empty for verbose build output RUNTEST_ARGS Arguments passed to runtest.py TEST_WAYS Testsuite ways to run @@ -573,10 +574,17 @@ function build_hadrian() { export XZ_OPT="${XZ_OPT:-} -T$cores" fi - case "${CROSS_STAGE:-2}" in + case "${FINAL_CROSS_STAGE:-2}" in 2) BINDIST_TARGET="binary-dist";; - 3) BINDIST_TARGET="binary-dist-stage3";; - *) fail "Unknown CROSS_STAGE, must be 2 or 3";; + # Stage2 bindists are (almost) a byproduct of Stage3 bindists. So, we + # bundle both of them when the Stage3 bindist is built. + 3) + BINDIST_TARGET="binary-dist binary-dist-stage3" + if [[ -z "${BIN_DIST_NAME_STAGE3:-}" ]]; then + fail "FINAL_CROSS_STAGE=3 requires BIN_DIST_NAME_STAGE3 to be set" + fi + ;; + *) fail "Unknown FINAL_CROSS_STAGE, must be 2 or 3";; esac if [[ -n "${REINSTALL_GHC:-}" ]]; then @@ -590,6 +598,9 @@ function build_hadrian() { *) run_hadrian test:all_deps $BINDIST_TARGET mv _build/bindist/ghc*.tar.xz "$BIN_DIST_NAME.tar.xz" + if [[ "${FINAL_CROSS_STAGE:-2}" == "3" ]]; then + mv _build/bindist-stage3/ghc*.tar.xz "$BIN_DIST_NAME_STAGE3.tar.xz" + fi ;; esac fi @@ -648,6 +659,62 @@ function install_bindist() { end_section install-bindist } +# Smoke-test the cross-compiled Stage2 and Stage2 bindists using the emulator. +# For cross jobs with a working emulator this runs instead of `test_hadrian` +# as a separate validation step after the build. +function smoke_test() { + check_msys2_deps _build/stage1/bin/ghc --version + check_release_build + + if [[ "${CROSS_EMULATOR:-}" == "NOT_SET" ]]; then + info "Cannot test cross-compiled build without CROSS_EMULATOR being set." + return + fi + + local instdir="$TOP/_build/install" + local test_compiler="$instdir/bin/${cross_prefix}ghc$exe" + install_bindist _build/bindist/ghc-*/ "$instdir" + echo 'main = putStrLn "hello world"' > expected + run "$test_compiler" -package ghc "$TOP/.gitlab/hello.hs" -o hello + + if [[ "${CROSS_TARGET:-no_cross_target}" =~ "mingw" ]]; then + ${CROSS_EMULATOR:-} ./hello.exe > actual + else + ${CROSS_EMULATOR:-} ./hello > actual + fi + + # We have to use `-w` to make the test more stable across supported + # platforms, i.e. Windows: + # $ cmp expected actual + # differ: byte 30, line 1 + # $ diff expected actual + # 1c1 + # < main = putStrLn "hello world" + # --- + # > main = putStrLn "hello world" + run diff -w expected actual + + if [[ "${FINAL_CROSS_STAGE:-2}" == "3" ]]; then + local stage3_dir + stage3_dir="$(echo _build/bindist-stage3/ghc-*/)" + local stage3_ghc="$stage3_dir/bin/ghc$exe" + + info "Smoke-testing stage3 compiler..." + file "$stage3_ghc" + run ${CROSS_EMULATOR} "$stage3_ghc" --info + + run ${CROSS_EMULATOR} "$stage3_ghc" -package ghc "$TOP/.gitlab/hello.hs" -o hello-stage3 + + if [[ "${CROSS_TARGET:-no_cross_target}" =~ "mingw" ]]; then + ${CROSS_EMULATOR:-} ./hello-stage3.exe > actual-stage3 + else + ${CROSS_EMULATOR:-} ./hello-stage3 > actual-stage3 + fi + + run diff -w expected actual-stage3 + fi +} + function test_hadrian() { check_msys2_deps _build/stage1/bin/ghc --version check_release_build @@ -672,35 +739,7 @@ function test_hadrian() { fi fi - - if [[ "${CROSS_EMULATOR:-}" == "NOT_SET" ]]; then - info "Cannot test cross-compiled build without CROSS_EMULATOR being set." - return - # If we have set CROSS_EMULATOR, then can't test using normal testsuite. - elif [ -n "${CROSS_EMULATOR:-}" ] && [[ "${CROSS_TARGET:-}" != *"wasm"* ]]; then - local instdir="$TOP/_build/install" - local test_compiler="$instdir/bin/${cross_prefix}ghc$exe" - install_bindist _build/bindist/ghc-*/ "$instdir" - echo 'main = putStrLn "hello world"' > expected - run "$test_compiler" -package ghc "$TOP/.gitlab/hello.hs" -o hello - - if [[ "${CROSS_TARGET:-no_cross_target}" =~ "mingw" ]]; then - ${CROSS_EMULATOR:-} ./hello.exe > actual - else - ${CROSS_EMULATOR:-} ./hello > actual - fi - - # We have to use `-w` to make the test more stable across supported - # platforms, i.e. Windows: - # $ cmp expected actual - # differ: byte 30, line 1 - # $ diff expected actual - # 1c1 - # < main = putStrLn "hello world" - # --- - # > main = putStrLn "hello world" - run diff -w expected actual - elif [[ -n "${REINSTALL_GHC:-}" ]]; then + if [[ -n "${REINSTALL_GHC:-}" ]]; then run_hadrian \ test \ --test-root-dirs=testsuite/tests/stage1 \ @@ -1131,6 +1170,7 @@ case ${1:-help} in time_it "test" test_hadrian || res=$? push_perf_notes exit $res ;; + smoke_test) time_it "smoke-test" smoke_test ;; run_hadrian) shift; run_hadrian "$@" ;; perf_test) run_perf_test ;; abi_test) abi_test ;; ===================================== .gitlab/generate-ci/gen_ci.hs ===================================== @@ -110,26 +110,49 @@ data Opsys = Linux LinuxDistro | Darwin | FreeBSD14 - | Windows deriving (Eq) + | Windows deriving (Eq, Show) data LinuxDistro = Debian13 | Debian12 - | Debian12Wine - | Debian13Riscv | Debian11 - | Debian11Js | Fedora43 - | Ubuntu2404LoongArch64 | Ubuntu2404 | Ubuntu2204 | Alpine312 | Alpine323 - | AlpineWasm | Rocky8 - deriving (Eq) - -data Arch = Amd64 | AArch64 | I386 + deriving (Eq, Show) + +data Arch = Amd64 | AArch64 | I386 | RiscV64 | LoongArch | Wasm32 | JavaScript + deriving (Show) + +-- | Describes what the docker image is for: the development flavour of the +-- build environment, orthogonal to the image's own architecture ('Arch'). +-- E.g. @FlavorRiscV@ denotes the image that carries the RISC-V cross +-- toolchain. +data ImageFlavor + = FlavorVanilla + | FlavorRiscV + | FlavorLoongArch + | FlavorWine + | FlavorEmscripten + | FlavorWasm + deriving (Eq, Show) + +-- | OS element of a configure target triple: this is the OS of the *target* +-- platform, not the build environment (that is 'Opsys'/'LinuxDistro'). +data TargetOs + = TargetOsLinuxGnu + | TargetOsGhcjs + | TargetOsMingw32 + | TargetOsWasi + deriving (Eq, Show) + +data TargetPlatform = TargetPlatform + { tpArch :: Arch -- ^ target architecture, e.g. 'RiscV64' + , tpOs :: TargetOs -- ^ target OS, e.g. 'TargetOsLinuxGnu' + } data BignumBackend = Native | Gmp deriving Eq @@ -158,8 +181,8 @@ data BuildConfig , withAssertions :: Bool , withNuma :: Bool , withZstd :: Bool - , crossTarget :: Maybe String - , crossStage :: Maybe Int + , crossTarget :: Maybe TargetPlatform + , crossStages :: Maybe CrossStages , crossEmulator :: CrossEmulator , configureWrapper :: Maybe String , fullyStatic :: Bool @@ -180,7 +203,7 @@ configureArgsStr bc = unwords $ ["--enable-unregisterised"| unregisterised bc ] ++ ["--disable-tables-next-to-code" | not (tablesNextToCode bc) ] ++ ["--with-intree-gmp" | Just _ <- [crossTarget bc] ] - ++ ["--with-system-libffi" | crossTarget bc == Just "wasm32-wasi" ] + ++ ["--with-system-libffi" | (tpOs <$> crossTarget bc) == Just TargetOsWasi ] ++ ["--enable-ipe-data-compression" | withZstd bc ] ++ ["--enable-strict-ghc-toolchain-check"] @@ -229,7 +252,7 @@ vanilla = BuildConfig , withNuma = False , withZstd = False , crossTarget = Nothing - , crossStage = Nothing + , crossStages = Nothing , crossEmulator = NoEmulator , configureWrapper = Nothing , fullyStatic = False @@ -275,17 +298,54 @@ static = vanilla { fullyStatic = True } staticNativeInt :: BuildConfig staticNativeInt = static { bignumBackend = Native } -crossConfig :: String -- ^ target triple +-- | The cross stages a build produces binary distributions for +data CrossStages + = -- | Builds a cross-compiler (build == host, host /= target): + -- its bindist runs on the build host. + Stage2 + | -- | Implies `Stage2` and additionally builds a cross-compiled compiler + -- (build /= host, host == target). + Stage2And3 + deriving stock (Eq, Ord) + +crossConfig :: TargetPlatform -- ^ target platform -> CrossEmulator -- ^ emulator for testing -> Maybe String -- ^ Configure wrapper + -> CrossStages -- ^ final cross stages to build bindists for -> BuildConfig -crossConfig triple emulator configure_wrapper = - vanilla { crossTarget = Just triple - , crossStage = Just 2 +crossConfig targetPlatform emulator configure_wrapper crossStage = + vanilla { crossTarget = Just targetPlatform + , crossStages = Just crossStage , crossEmulator = emulator , configureWrapper = configure_wrapper } +-- | The docker image flavour of a build, derived from its cross target +-- (the image must carry the target's toolchain) or 'FlavorVanilla' for +-- plain builds. +buildFlavor :: BuildConfig -> ImageFlavor +buildFlavor bc = maybe FlavorVanilla flavorOf (crossTarget bc) + where + flavorOf :: TargetPlatform -> ImageFlavor + flavorOf TargetPlatform{ tpArch = arch, tpOs = os } = case (arch, os) of + (AArch64, TargetOsLinuxGnu) -> FlavorVanilla + (AArch64, TargetOsMingw32) -> FlavorWine + (RiscV64, TargetOsLinuxGnu) -> FlavorRiscV + (LoongArch, TargetOsLinuxGnu) -> FlavorLoongArch + (Wasm32, TargetOsWasi) -> FlavorWasm + (JavaScript, TargetOsGhcjs) -> FlavorEmscripten + _ -> error $ + "flavorOf: no image flavour for target " ++ show arch ++ "/" ++ show os + +-- | The configure target triple the build system consumes, e.g. +-- @"riscv64-linux-gnu"@. +renderTriple :: TargetPlatform -> String +renderTriple TargetPlatform{ tpArch = arch, tpOs = os } = case os of + TargetOsLinuxGnu -> archName arch ++ "-linux-gnu" + TargetOsGhcjs -> archName arch ++ "-unknown-ghcjs" + TargetOsMingw32 -> archName arch ++ "-unknown-mingw32" + TargetOsWasi -> archName arch ++ "-wasi" + llvm :: BuildConfig llvm = vanilla { llvmBootstrap = True, testsuiteWays = ["llvm", "optllvm"] } @@ -307,16 +367,16 @@ usePerfProfilingTestsuite bc = bc { testsuiteUsePerf = True } -- | These tags have to match what we call the runners on gitlab runnerTag :: Arch -> Opsys -> String -runnerTag arch (Linux _) = - case arch of - Amd64 -> "x86_64-linux" - AArch64 -> "aarch64-linux" - I386 -> "x86_64-linux" -runnerTag AArch64 Darwin = "aarch64-darwin" -runnerTag Amd64 Darwin = "x86_64-darwin-m1" -runnerTag Amd64 Windows = "new-x86_64-windows" -runnerTag Amd64 FreeBSD14 = "x86_64-freebsd14" -runnerTag _ _ = error "Invalid arch/opsys" +runnerTag arch opsys = case (arch, opsys) of + (Amd64, Linux{}) -> "x86_64-linux" + (AArch64, Linux{}) -> "aarch64-linux" + (I386, Linux{}) -> "x86_64-linux" + (AArch64, Darwin) -> "aarch64-darwin" + (Amd64, Darwin) -> "x86_64-darwin-m1" + (Amd64, Windows) -> "new-x86_64-windows" + (Amd64, FreeBSD14) -> "x86_64-freebsd14" + _ -> error $ "runnerTag: no runner for " + ++ show arch ++ "/" ++ show opsys tags :: Arch -> Opsys -> BuildConfig -> [String] tags arch opsys _bc = [runnerTag arch opsys] -- Tag for which runners we can use @@ -324,52 +384,92 @@ tags arch opsys _bc = [runnerTag arch opsys] -- Tag for which runners we can use runnerPerfTag :: Arch -> Opsys -> String runnerPerfTag arch sys = runnerTag arch sys ++ "-perf" --- These names are used to find the docker image so they have to match what is --- in the docker registry. +-- | These names are used to find the docker image so they have to match what +-- is in the docker registry. distroName :: LinuxDistro -> String distroName Debian13 = "deb13" distroName Debian12 = "deb12" distroName Debian11 = "deb11" -distroName Debian11Js = "deb11-emsdk-closure" -distroName Debian13Riscv = "deb13-riscv" -distroName Debian12Wine = "deb12-wine" distroName Fedora43 = "fedora43" -distroName Ubuntu2404LoongArch64 = "ubuntu24_04-loongarch" distroName Ubuntu2204 = "ubuntu22_04" distroName Ubuntu2404 = "ubuntu24_04" distroName Alpine312 = "alpine3_12" distroName Alpine323 = "alpine3_23" -distroName AlpineWasm = "alpine3_23-wasm" distroName Rocky8 = "rocky8" -opsysName :: Opsys -> String -opsysName (Linux distro) = "linux-" ++ distroName distro -opsysName Darwin = "darwin" -opsysName FreeBSD14 = "freebsd14" -opsysName Windows = "windows" +-- | The distro tag used in docker image names, extended with the toolchain +-- suffix of the image flavour (e.g. 'Debian13'' + 'FlavorRiscV' -> +-- @"deb13-riscv"@). +distroImageTag :: LinuxDistro -> ImageFlavor -> String +distroImageTag distro flavor = distroName distro ++ flavorSuffix flavor + where + flavorSuffix :: ImageFlavor -> String + flavorSuffix FlavorVanilla = "" + flavorSuffix FlavorRiscV = "-riscv" + flavorSuffix FlavorLoongArch = "-loongarch" + flavorSuffix FlavorWine = "-wine" + flavorSuffix FlavorEmscripten = "-emsdk-closure" + flavorSuffix FlavorWasm = "-wasm" + +-- | OS name as it appears for job identity and docker image tags, qualified +-- by the working image flavour if any (e.g. @"linux-deb13-riscv"@). +opsysNameWithFlavor :: Opsys -> ImageFlavor -> String +opsysNameWithFlavor (Linux distro) flavor = "linux-" ++ distroImageTag distro flavor +opsysNameWithFlavor opsys _ = opsysName opsys + where + opsysName :: Opsys -> String + opsysName (Linux distro) = "linux-" ++ distroName distro + opsysName Darwin = "darwin" + opsysName FreeBSD14 = "freebsd14" + opsysName Windows = "windows" archName :: Arch -> String -archName Amd64 = "x86_64" -archName AArch64 = "aarch64" -archName I386 = "i386" +archName Amd64 = "x86_64" +archName AArch64 = "aarch64" +archName I386 = "i386" +archName RiscV64 = "riscv64" +archName LoongArch = "loongarch64" +archName Wasm32 = "wasm32" +archName JavaScript = "javascript" binDistName :: Arch -> Opsys -> BuildConfig -> String -binDistName arch opsys bc = "ghc-" ++ testEnv arch opsys bc +binDistName arch opsys bc = "ghc-" ++ testEnvWith (archName arch) opsys bc + +-- | Name of the stage3 (target-platform) bindist. +-- +-- Because that bindist lives on its target, it is named as if built natively +-- on the target: target architecture, the OS of the job, and no +-- cross/toolchain flavour markers. +stage3BinDistName :: Opsys -> BuildConfig -> Maybe String +stage3BinDistName opsys bc + | buildsTargetBindist + , Just TargetPlatform{ tpArch = arch } <- crossTarget bc + = Just $ binDistName arch opsys (bc { crossTarget = Nothing }) + | otherwise + = Nothing + where + -- Whether any bindist produced by this build configuration runs on its + -- target platform rather than on the build host. + buildsTargetBindist :: Bool + buildsTargetBindist = crossStages bc == Just Stage2And3 -- | Test env should create a string which changes whenever the 'BuildConfig' changes. -- Either the change is reflected by modifying the flavourString or directly (as is -- the case for settings which affect environment variables) testEnv :: Arch -> Opsys -> BuildConfig -> String -testEnv arch opsys bc = +testEnv arch = testEnvWith (archName arch) + +testEnvWith :: String -> Opsys -> BuildConfig -> String +testEnvWith archN opsys bc = intercalate "-" $ concat - [ [ archName arch - , opsysName opsys ] + [ [ archN + , opsysNameWithFlavor opsys (buildFlavor bc) ] , ["int_" ++ bignumString (bignumBackend bc) | bignumBackend bc /= Gmp] , ["unreg" | unregisterised bc ] , ["numa" | withNuma bc ] , ["zstd" | withZstd bc ] , ["no_tntc" | not (tablesNextToCode bc) ] - , ["cross_"++triple | Just triple <- pure $ crossTarget bc ] + , ["cross_"++renderTriple ct | Just ct <- pure $ crossTarget bc ] , [flavourString (mkJobFlavour bc)] ] @@ -392,18 +492,18 @@ flavourString (Flavour base trans) = base_string base ++ concatMap (("+" ++) . f flavour_string TextWithSIMDUTF = "text_simdutf" -- The path to the docker image (just for linux builders) -dockerImage :: Arch -> Opsys -> Maybe String -dockerImage arch (Linux distro) = +dockerImage :: Arch -> Opsys -> BuildConfig -> Maybe String +dockerImage arch (Linux distro) bc = Just image where image = mconcat [ "registry.gitlab.haskell.org/ghc/ci-images/" , archName arch , "-linux-" - , distroName distro + , distroImageTag distro (buildFlavor bc) , ":$DOCKER_REV" ] -dockerImage _ _ = Nothing +dockerImage _ _ _ = Nothing ----------------------------------------------------------------------------- -- Platform-specific variables @@ -444,8 +544,8 @@ brokenTest :: TestName -- ^ test name -> Variables brokenTest test _why = "BROKEN_TESTS" =: test -opsysVariables :: Arch -> Opsys -> Variables -opsysVariables _ FreeBSD14 = mconcat +opsysVariables :: Arch -> Opsys -> ImageFlavor -> Variables +opsysVariables _ FreeBSD14 _ = mconcat [ -- N.B. we use iconv from ports as I see linker errors when we attempt -- to use the "native" iconv embedded in libc as suggested by the -- porting guide [1]. @@ -459,8 +559,8 @@ opsysVariables _ FreeBSD14 = mconcat , "FETCH_GHC_VERSION" =: "9.10.3" , "CABAL_INSTALL_VERSION" =: "3.14.2.0" ] -opsysVariables arch (Linux distro) = distroVariables arch distro -opsysVariables AArch64 (Darwin {}) = mconcat +opsysVariables arch (Linux distro) flavor = distroVariables arch distro flavor +opsysVariables AArch64 (Darwin {}) _ = mconcat [ "NIX_SYSTEM" =: "aarch64-darwin" , "MACOSX_DEPLOYMENT_TARGET" =: "11.0" , "LANG" =: "en_US.UTF-8" @@ -468,7 +568,7 @@ opsysVariables AArch64 (Darwin {}) = mconcat -- Fonts can't be installed on darwin , "HADRIAN_ARGS" =: "--docs=no-sphinx-pdfs" ] -opsysVariables Amd64 (Darwin {}) = mconcat +opsysVariables Amd64 (Darwin {}) _ = mconcat [ "NIX_SYSTEM" =: "x86_64-darwin" , "MACOSX_DEPLOYMENT_TARGET" =: "11.0" -- Only Sierra and onwards supports clock_gettime. See #12858 @@ -481,16 +581,15 @@ opsysVariables Amd64 (Darwin {}) = mconcat , "CONFIGURE_ARGS" =: "--with-intree-gmp --with-system-libffi" -- Fonts can't be installed on darwin , "HADRIAN_ARGS" =: "--docs=no-sphinx-pdfs" - ] -opsysVariables _ (Windows {}) = mconcat +opsysVariables _ (Windows {}) _ = mconcat [ "MSYSTEM" =: "CLANG64" , "LANG" =: "en_US.UTF-8" , "CABAL_INSTALL_VERSION" =: "3.14.2.0" , "HADRIAN_ARGS" =: "--docs=no-sphinx-pdfs" , "FETCH_GHC_VERSION" =: "9.10.3" ] -opsysVariables _ _ = mempty +opsysVariables _ _ _ = mempty alpineVariables :: Arch -> Variables alpineVariables arch = mconcat $ @@ -509,10 +608,13 @@ alpineVariables arch = mconcat $ ] -distroVariables :: Arch -> LinuxDistro -> Variables -distroVariables arch Alpine312 = alpineVariables arch -distroVariables arch Alpine323 = alpineVariables arch -distroVariables _ _ = mempty +distroVariables :: Arch -> LinuxDistro -> ImageFlavor -> Variables +distroVariables arch Alpine312 _ = alpineVariables arch +-- The wasm image runs its own (libc-free) toolchain, so the alpine broken +-- tests do not apply to it. +distroVariables _ Alpine323 FlavorWasm = mempty +distroVariables arch Alpine323 _ = alpineVariables arch +distroVariables _ _ _ = mempty ----------------------------------------------------------------------------- -- Cache settings, what to cache and when can we share the cache @@ -524,8 +626,8 @@ data Cache } -- The cache doesn't depend on the BuildConfig because we only cache the cabal store. -mkCacheKey :: Arch -> Opsys -> String -mkCacheKey arch opsys = archName arch <> "-" <> opsysName opsys <> "-$CACHE_REV" +mkCacheKey :: Arch -> Opsys -> ImageFlavor -> String +mkCacheKey arch opsys flavor = archName arch <> "-" <> opsysNameWithFlavor opsys flavor <> "-$CACHE_REV" instance ToJSON Cache where toJSON Cache {..} = object @@ -807,7 +909,7 @@ data Job , jobArtifacts :: Artifacts , jobCache :: Cache , jobRules :: OnOffRules - , jobPlatform :: (Arch, Opsys) + , jobPlatform :: (Arch, Opsys, ImageFlavor) } instance Show Job where @@ -837,7 +939,7 @@ instance ToJSON Job where job :: Arch -> Opsys -> BuildConfig -> NamedJob Job job arch opsys buildConfig = NamedJob { name = jobName, jobInfo = Job {..} } where - jobPlatform = (arch, opsys) + jobPlatform = (arch, opsys, buildFlavor buildConfig) jobRules = emptyRules jobName @@ -845,14 +947,14 @@ job arch opsys buildConfig = NamedJob { name = jobName, jobInfo = Job {..} } jobTags = tags arch opsys buildConfig - jobDockerImage = dockerImage arch opsys + jobDockerImage = dockerImage arch opsys buildConfig jobScript | Windows <- opsys = [ "bash .gitlab/ci.sh setup" , "bash .gitlab/ci.sh configure" , "bash .gitlab/ci.sh build_hadrian" - , "bash .gitlab/ci.sh test_hadrian" + , "bash " ++ testStep ] | otherwise = [ "find libraries -name config.sub -exec cp config.sub {} \\;" | Darwin == opsys ] ++ @@ -860,9 +962,16 @@ job arch opsys buildConfig = NamedJob { name = jobName, jobInfo = Job {..} } [ ".gitlab/ci.sh setup" , ".gitlab/ci.sh configure" , ".gitlab/ci.sh build_hadrian" - , ".gitlab/ci.sh test_hadrian" + , testStep ] + -- Cross jobs with a working emulator smoke-test the bindist under the + -- emulator as a separate step after the build; everything else runs the + -- testsuite. + testStep + | Emulator _ <- crossEmulator buildConfig = ".gitlab/ci.sh smoke_test" + | otherwise = ".gitlab/ci.sh test_hadrian" + jobAfterScript | Windows <- opsys = [ "bash .gitlab/ci.sh save_cache" @@ -880,16 +989,20 @@ job arch opsys buildConfig = NamedJob { name = jobName, jobInfo = Job {..} } jobDependencies = [] jobVariables = mconcat - [ opsysVariables arch opsys + [ opsysVariables arch opsys (buildFlavor buildConfig) , "TEST_ENV" =: testEnv arch opsys buildConfig , "BIN_DIST_NAME" =: binDistName arch opsys buildConfig + , maybe mempty ("BIN_DIST_NAME_STAGE3" =:) (stage3BinDistName opsys buildConfig) , "BUILD_FLAVOUR" =: flavourString jobFlavour , "BIGNUM_BACKEND" =: bignumString (bignumBackend buildConfig) , "CONFIGURE_ARGS" =: configureArgsStr buildConfig , "INSTALL_CONFIGURE_ARGS" =: "--enable-strict-ghc-toolchain-check" , maybe mempty ("CONFIGURE_WRAPPER" =:) (configureWrapper buildConfig) - , maybe mempty ("CROSS_TARGET" =:) (crossTarget buildConfig) - , maybe mempty (("CROSS_STAGE" =:) . show) (crossStage buildConfig) + , maybe mempty (("CROSS_TARGET" =:) . renderTriple) (crossTarget buildConfig) + , case crossStages buildConfig of + Nothing -> mempty + Just Stage2 -> "FINAL_CROSS_STAGE" =: "2" + Just Stage2And3 -> "FINAL_CROSS_STAGE" =: "3" , case crossEmulator buildConfig of NoEmulator -- we need an emulator but it isn't set. Won't run the testsuite @@ -922,6 +1035,8 @@ job arch opsys buildConfig = NamedJob { name = jobName, jobInfo = Job {..} } trim :: String -> String trim = dropWhileEnd isSpace . dropWhile isSpace + stage3Artifacts = maybe [] (\n -> [n ++ ".tar.xz"]) (stage3BinDistName opsys buildConfig) + -- Keep in sync with the exclude list in `function clean()` in -- `.gitlab/ci.sh`! jobArtifacts = Artifacts @@ -930,6 +1045,7 @@ job arch opsys buildConfig = NamedJob { name = jobName, jobInfo = Job {..} } , artifactPaths = [binDistName arch opsys buildConfig ++ ".tar.xz" ,"junit.xml" ,"unexpected-test-output.tar.gz"] + ++ stage3Artifacts , artifactsWhen = ArtifactsAlways } @@ -940,7 +1056,7 @@ job arch opsys buildConfig = NamedJob { name = jobName, jobInfo = Job {..} } Cache { cachePaths = [], cacheKey = "no-caching" } | otherwise = Cache { cachePaths = [ "cabal-cache", "toolchain" ] - , cacheKey = mkCacheKey arch opsys + , cacheKey = mkCacheKey arch opsys (buildFlavor buildConfig) } @@ -1163,7 +1279,6 @@ debian_x86 = , disableValidate (standardBuilds Amd64 (Linux Debian12)) , disableValidate (standardBuilds Amd64 (Linux Debian13)) - -- Validate only builds , fastCI (validateBuilds Amd64 (Linux validate_debian) debug) , validateBuilds Amd64 (Linux validate_debian) nativeInt @@ -1288,16 +1403,17 @@ alpine_aarch64 = [ cross_jobs :: [JobGroup Job] cross_jobs = [ -- x86 -> aarch64 - validateBuilds Amd64 (Linux Debian13) (crossConfig "aarch64-linux-gnu" (Emulator "qemu-aarch64 -L /usr/aarch64-linux-gnu") Nothing) + validateBuilds Amd64 (Linux Debian13) (crossConfig (TargetPlatform AArch64 TargetOsLinuxGnu) (Emulator "qemu-aarch64 -L /usr/aarch64-linux-gnu") Nothing Stage2) - -- x86_64 -> riscv - , addValidateRule RiscV (validateBuilds Amd64 (Linux Debian13Riscv) (crossConfig "riscv64-linux-gnu" (Emulator "qemu-riscv64 -L /usr/riscv64-linux-gnu") Nothing)) + -- Stage2: x86_64 (build/host) -> riscv64 (target) + -- Stage3: x86_64 (build) -> riscv64 (host/target) + , addValidateRule RiscV (validateBuilds Amd64 (Linux Debian13) (crossConfig (TargetPlatform RiscV64 TargetOsLinuxGnu) (Emulator "qemu-riscv64 -L /usr/riscv64-linux-gnu") Nothing Stage2And3)) -- x86_64 -> loongarch64 - , addValidateRule LoongArch64 (validateBuilds Amd64 (Linux Ubuntu2404LoongArch64) (crossConfig "loongarch64-linux-gnu" (Emulator "qemu-loongarch64 -L /usr/loongarch64-linux-gnu") Nothing)) + , addValidateRule LoongArch64 (validateBuilds Amd64 (Linux Ubuntu2404) (crossConfig (TargetPlatform LoongArch TargetOsLinuxGnu) (Emulator "qemu-loongarch64 -L /usr/loongarch64-linux-gnu") Nothing Stage2)) -- Javascript - , addValidateRule JSBackend (validateBuilds Amd64 (Linux Debian11Js) javascriptConfig) + , addValidateRule JSBackend (validateBuilds Amd64 (Linux Debian11) javascriptConfig) -- Wasm , make_wasm_jobs wasm_build_config @@ -1309,13 +1425,13 @@ cross_jobs = [ -- Linux Aarch64 (Wine + FEX + MSYS64) => Windows Aarch64 , makeWinArmJobs $ addValidateRule WinArm64 - (validateBuilds AArch64 (Linux Debian12Wine) winAarch64Config) + (validateBuilds AArch64 (Linux Debian12) winAarch64Config) , makeWinArmJobs $ addValidateRule WinArm64LLVM - (validateBuilds AArch64 (Linux Debian12Wine) (winAarch64Config {llvmBootstrap = True})) + (validateBuilds AArch64 (Linux Debian12) (winAarch64Config {llvmBootstrap = True})) ] where - javascriptConfig = (crossConfig "javascript-unknown-ghcjs" (NoEmulatorNeeded TimeoutIncrease) (Just "emconfigure")) + javascriptConfig = (crossConfig (TargetPlatform JavaScript TargetOsGhcjs) (NoEmulatorNeeded TimeoutIncrease) (Just "emconfigure") Stage2) { bignumBackend = Native } makeWinArmJobs = modifyJobs @@ -1354,7 +1470,7 @@ cross_jobs = [ llvm_prefix = "/opt/llvm-mingw-linux/bin/aarch64-w64-mingw32-" cflags = "-fuse-ld=" ++ llvm_prefix ++ "ld --rtlib=compiler-rt" - winAarch64Config = (crossConfig "aarch64-unknown-mingw32" (Emulator "/opt/wine-arm64ec-msys2-deb12/bin/wine") Nothing) + winAarch64Config = (crossConfig (TargetPlatform AArch64 TargetOsMingw32) (Emulator "/opt/wine-arm64ec-msys2-deb12/bin/wine") Nothing Stage2) { bignumBackend = Native } make_wasm_jobs cfg = @@ -1364,10 +1480,10 @@ cross_jobs = [ . setVariable "HADRIAN_ARGS" "--docs=no-sphinx-pdfs --docs=no-sphinx-man" . delVariable "INSTALL_CONFIGURE_ARGS" ) - $ addValidateRule WasmBackend $ validateBuilds Amd64 (Linux AlpineWasm) cfg + $ addValidateRule WasmBackend $ validateBuilds Amd64 (Linux Alpine323) cfg wasm_build_config = - (crossConfig "wasm32-wasi" (NoEmulatorNeeded NoTimeoutIncrease) Nothing) + (crossConfig (TargetPlatform Wasm32 TargetOsWasi) (NoEmulatorNeeded NoTimeoutIncrease) Nothing Stage2) { hostFullyStatic = True , buildFlavour = Release -- TODO: This needs to be validate but wasm backend doesn't pass yet , textWithSIMDUTF = True @@ -1392,8 +1508,8 @@ job_groups = -- Platform mapping for GHCup metadata --------------------------------------------------------------------- -mkPlatform :: Arch -> Opsys -> String -mkPlatform arch opsys = archName arch <> "-" <> opsysName opsys +mkPlatform :: (Arch, Opsys, ImageFlavor) -> String +mkPlatform (arch, opsys, flavor) = archName arch <> "-" <> opsysNameWithFlavor opsys flavor -- | This map tells us for a specific arch/opsys combo what the job name for -- nightly/release pipelines is. This is used by the ghcup metadata generation so that @@ -1438,7 +1554,7 @@ platform_mapping = Map.map go combined_result process sel = Map.fromListWith combine - [ (uncurry mkPlatform (jobPlatform (jobInfo j)), j) + [ (mkPlatform (jobPlatform (jobInfo j)), j) | (sel -> Just j) <- job_groups ] ===================================== .gitlab/jobs.yaml ===================================== @@ -303,7 +303,7 @@ ".gitlab/ci.sh setup", ".gitlab/ci.sh configure", ".gitlab/ci.sh build_hadrian", - ".gitlab/ci.sh test_hadrian" + ".gitlab/ci.sh smoke_test" ], "stage": "full-build", "tags": [ @@ -319,9 +319,9 @@ "CONFIGURE_ARGS": "--with-intree-gmp --enable-strict-ghc-toolchain-check", "CONF_CC_OPTS_STAGE2": "-fuse-ld=/opt/llvm-mingw-linux/bin/aarch64-w64-mingw32-ld --rtlib=compiler-rt", "CROSS_EMULATOR": "/opt/wine-arm64ec-msys2-deb12/bin/wine", - "CROSS_STAGE": "2", "CROSS_TARGET": "aarch64-unknown-mingw32", "CXX": "/opt/llvm-mingw-linux/bin/aarch64-w64-mingw32-clang++", + "FINAL_CROSS_STAGE": "2", "HADRIAN_ARGS": "--docs=none", "INSTALL_CONFIGURE_ARGS": "--enable-strict-ghc-toolchain-check", "LD": "/opt/llvm-mingw-linux/bin/aarch64-w64-mingw32-ld", @@ -387,7 +387,7 @@ ".gitlab/ci.sh setup", ".gitlab/ci.sh configure", ".gitlab/ci.sh build_hadrian", - ".gitlab/ci.sh test_hadrian" + ".gitlab/ci.sh smoke_test" ], "stage": "full-build", "tags": [ @@ -403,9 +403,9 @@ "CONFIGURE_ARGS": "--with-intree-gmp --enable-strict-ghc-toolchain-check", "CONF_CC_OPTS_STAGE2": "-fuse-ld=/opt/llvm-mingw-linux/bin/aarch64-w64-mingw32-ld --rtlib=compiler-rt", "CROSS_EMULATOR": "/opt/wine-arm64ec-msys2-deb12/bin/wine", - "CROSS_STAGE": "2", "CROSS_TARGET": "aarch64-unknown-mingw32", "CXX": "/opt/llvm-mingw-linux/bin/aarch64-w64-mingw32-clang++", + "FINAL_CROSS_STAGE": "2", "HADRIAN_ARGS": "--docs=none", "INSTALL_CONFIGURE_ARGS": "--enable-strict-ghc-toolchain-check", "LD": "/opt/llvm-mingw-linux/bin/aarch64-w64-mingw32-ld", @@ -1111,7 +1111,7 @@ ".gitlab/ci.sh setup", ".gitlab/ci.sh configure", ".gitlab/ci.sh build_hadrian", - ".gitlab/ci.sh test_hadrian" + ".gitlab/ci.sh smoke_test" ], "stage": "full-build", "tags": [ @@ -1127,9 +1127,9 @@ "CONFIGURE_ARGS": "--with-intree-gmp --enable-strict-ghc-toolchain-check", "CONF_CC_OPTS_STAGE2": "-fuse-ld=/opt/llvm-mingw-linux/bin/aarch64-w64-mingw32-ld --rtlib=compiler-rt", "CROSS_EMULATOR": "/opt/wine-arm64ec-msys2-deb12/bin/wine", - "CROSS_STAGE": "2", "CROSS_TARGET": "aarch64-unknown-mingw32", "CXX": "/opt/llvm-mingw-linux/bin/aarch64-w64-mingw32-clang++", + "FINAL_CROSS_STAGE": "2", "HADRIAN_ARGS": "--docs=none", "INSTALL_CONFIGURE_ARGS": "--enable-strict-ghc-toolchain-check", "LD": "/opt/llvm-mingw-linux/bin/aarch64-w64-mingw32-ld", @@ -1196,7 +1196,7 @@ ".gitlab/ci.sh setup", ".gitlab/ci.sh configure", ".gitlab/ci.sh build_hadrian", - ".gitlab/ci.sh test_hadrian" + ".gitlab/ci.sh smoke_test" ], "stage": "full-build", "tags": [ @@ -1212,9 +1212,9 @@ "CONFIGURE_ARGS": "--with-intree-gmp --enable-strict-ghc-toolchain-check", "CONF_CC_OPTS_STAGE2": "-fuse-ld=/opt/llvm-mingw-linux/bin/aarch64-w64-mingw32-ld --rtlib=compiler-rt", "CROSS_EMULATOR": "/opt/wine-arm64ec-msys2-deb12/bin/wine", - "CROSS_STAGE": "2", "CROSS_TARGET": "aarch64-unknown-mingw32", "CXX": "/opt/llvm-mingw-linux/bin/aarch64-w64-mingw32-clang++", + "FINAL_CROSS_STAGE": "2", "HADRIAN_ARGS": "--docs=none", "INSTALL_CONFIGURE_ARGS": "--enable-strict-ghc-toolchain-check", "LD": "/opt/llvm-mingw-linux/bin/aarch64-w64-mingw32-ld", @@ -2010,8 +2010,8 @@ "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_23-wasm-cross_wasm32-wasi-release+host_fully_static+text_simdutf", "BUILD_FLAVOUR": "release+host_fully_static+text_simdutf", "CONFIGURE_ARGS": "--with-intree-gmp --with-system-libffi --enable-strict-ghc-toolchain-check", - "CROSS_STAGE": "2", "CROSS_TARGET": "wasm32-wasi", + "FINAL_CROSS_STAGE": "2", "FIREFOX_LAUNCH_OPTS": "{\"browser\":\"firefox\",\"executablePath\":\"/usr/bin/firefox\"}", "HADRIAN_ARGS": "--docs=no-sphinx-pdfs --docs=no-sphinx-man", "RUNTEST_ARGS": "", @@ -2077,8 +2077,8 @@ "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_23-wasm-int_native-cross_wasm32-wasi-release+host_fully_static+text_simdutf", "BUILD_FLAVOUR": "release+host_fully_static+text_simdutf", "CONFIGURE_ARGS": "--with-intree-gmp --with-system-libffi --enable-strict-ghc-toolchain-check", - "CROSS_STAGE": "2", "CROSS_TARGET": "wasm32-wasi", + "FINAL_CROSS_STAGE": "2", "FIREFOX_LAUNCH_OPTS": "{\"browser\":\"firefox\",\"executablePath\":\"/usr/bin/firefox\"}", "HADRIAN_ARGS": "--docs=no-sphinx-pdfs --docs=no-sphinx-man", "RUNTEST_ARGS": "", @@ -2144,8 +2144,8 @@ "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_23-wasm-unreg-cross_wasm32-wasi-release+host_fully_static+text_simdutf", "BUILD_FLAVOUR": "release+host_fully_static+text_simdutf", "CONFIGURE_ARGS": "--enable-unregisterised --with-intree-gmp --with-system-libffi --enable-strict-ghc-toolchain-check", - "CROSS_STAGE": "2", "CROSS_TARGET": "wasm32-wasi", + "FINAL_CROSS_STAGE": "2", "FIREFOX_LAUNCH_OPTS": "{\"browser\":\"firefox\",\"executablePath\":\"/usr/bin/firefox\"}", "HADRIAN_ARGS": "--docs=no-sphinx-pdfs --docs=no-sphinx-man", "RUNTEST_ARGS": "", @@ -2212,8 +2212,8 @@ "BUILD_FLAVOUR": "validate", "CONFIGURE_ARGS": "--with-intree-gmp --enable-strict-ghc-toolchain-check", "CONFIGURE_WRAPPER": "emconfigure", - "CROSS_STAGE": "2", "CROSS_TARGET": "javascript-unknown-ghcjs", + "FINAL_CROSS_STAGE": "2", "INSTALL_CONFIGURE_ARGS": "--enable-strict-ghc-toolchain-check", "RUNTEST_ARGS": "-e config.timeout=900", "TEST_ENV": "x86_64-linux-deb11-emsdk-closure-int_native-cross_javascript-unknown-ghcjs-validate", @@ -2459,7 +2459,7 @@ ".gitlab/ci.sh setup", ".gitlab/ci.sh configure", ".gitlab/ci.sh build_hadrian", - ".gitlab/ci.sh test_hadrian" + ".gitlab/ci.sh smoke_test" ], "stage": "full-build", "tags": [ @@ -2471,8 +2471,8 @@ "BUILD_FLAVOUR": "validate", "CONFIGURE_ARGS": "--with-intree-gmp --enable-strict-ghc-toolchain-check", "CROSS_EMULATOR": "qemu-aarch64 -L /usr/aarch64-linux-gnu", - "CROSS_STAGE": "2", "CROSS_TARGET": "aarch64-linux-gnu", + "FINAL_CROSS_STAGE": "2", "INSTALL_CONFIGURE_ARGS": "--enable-strict-ghc-toolchain-check", "RUNTEST_ARGS": "-e config.timeout=900", "TEST_ENV": "x86_64-linux-deb13-cross_aarch64-linux-gnu-validate", @@ -2750,7 +2750,8 @@ "paths": [ "ghc-x86_64-linux-deb13-riscv-cross_riscv64-linux-gnu-validate.tar.xz", "junit.xml", - "unexpected-test-output.tar.gz" + "unexpected-test-output.tar.gz", + "ghc-riscv64-linux-deb13-validate.tar.xz" ], "reports": { "junit": "junit.xml" @@ -2783,7 +2784,7 @@ ".gitlab/ci.sh setup", ".gitlab/ci.sh configure", ".gitlab/ci.sh build_hadrian", - ".gitlab/ci.sh test_hadrian" + ".gitlab/ci.sh smoke_test" ], "stage": "full-build", "tags": [ @@ -2792,11 +2793,12 @@ "variables": { "BIGNUM_BACKEND": "gmp", "BIN_DIST_NAME": "ghc-x86_64-linux-deb13-riscv-cross_riscv64-linux-gnu-validate", + "BIN_DIST_NAME_STAGE3": "ghc-riscv64-linux-deb13-validate", "BUILD_FLAVOUR": "validate", "CONFIGURE_ARGS": "--with-intree-gmp --enable-strict-ghc-toolchain-check", "CROSS_EMULATOR": "qemu-riscv64 -L /usr/riscv64-linux-gnu", - "CROSS_STAGE": "2", "CROSS_TARGET": "riscv64-linux-gnu", + "FINAL_CROSS_STAGE": "3", "INSTALL_CONFIGURE_ARGS": "--enable-strict-ghc-toolchain-check", "RUNTEST_ARGS": "-e config.timeout=900", "TEST_ENV": "x86_64-linux-deb13-riscv-cross_riscv64-linux-gnu-validate", @@ -3687,7 +3689,7 @@ ".gitlab/ci.sh setup", ".gitlab/ci.sh configure", ".gitlab/ci.sh build_hadrian", - ".gitlab/ci.sh test_hadrian" + ".gitlab/ci.sh smoke_test" ], "stage": "full-build", "tags": [ @@ -3699,8 +3701,8 @@ "BUILD_FLAVOUR": "validate", "CONFIGURE_ARGS": "--with-intree-gmp --enable-strict-ghc-toolchain-check", "CROSS_EMULATOR": "qemu-loongarch64 -L /usr/loongarch64-linux-gnu", - "CROSS_STAGE": "2", "CROSS_TARGET": "loongarch64-linux-gnu", + "FINAL_CROSS_STAGE": "2", "INSTALL_CONFIGURE_ARGS": "--enable-strict-ghc-toolchain-check", "RUNTEST_ARGS": "-e config.timeout=900", "TEST_ENV": "x86_64-linux-ubuntu24_04-loongarch-cross_loongarch64-linux-gnu-validate", @@ -5995,8 +5997,8 @@ "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_23-wasm-cross_wasm32-wasi-release+host_fully_static+text_simdutf", "BUILD_FLAVOUR": "release+host_fully_static+text_simdutf", "CONFIGURE_ARGS": "--with-intree-gmp --with-system-libffi --enable-strict-ghc-toolchain-check", - "CROSS_STAGE": "2", "CROSS_TARGET": "wasm32-wasi", + "FINAL_CROSS_STAGE": "2", "FIREFOX_LAUNCH_OPTS": "{\"browser\":\"firefox\",\"executablePath\":\"/usr/bin/firefox\"}", "HADRIAN_ARGS": "--docs=no-sphinx-pdfs --docs=no-sphinx-man", "RUNTEST_ARGS": "", @@ -6062,8 +6064,8 @@ "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_23-wasm-int_native-cross_wasm32-wasi-release+host_fully_static+text_simdutf", "BUILD_FLAVOUR": "release+host_fully_static+text_simdutf", "CONFIGURE_ARGS": "--with-intree-gmp --with-system-libffi --enable-strict-ghc-toolchain-check", - "CROSS_STAGE": "2", "CROSS_TARGET": "wasm32-wasi", + "FINAL_CROSS_STAGE": "2", "FIREFOX_LAUNCH_OPTS": "{\"browser\":\"firefox\",\"executablePath\":\"/usr/bin/firefox\"}", "HADRIAN_ARGS": "--docs=no-sphinx-pdfs --docs=no-sphinx-man", "RUNTEST_ARGS": "", @@ -6129,8 +6131,8 @@ "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_23-wasm-unreg-cross_wasm32-wasi-release+host_fully_static+text_simdutf", "BUILD_FLAVOUR": "release+host_fully_static+text_simdutf", "CONFIGURE_ARGS": "--enable-unregisterised --with-intree-gmp --with-system-libffi --enable-strict-ghc-toolchain-check", - "CROSS_STAGE": "2", "CROSS_TARGET": "wasm32-wasi", + "FINAL_CROSS_STAGE": "2", "FIREFOX_LAUNCH_OPTS": "{\"browser\":\"firefox\",\"executablePath\":\"/usr/bin/firefox\"}", "HADRIAN_ARGS": "--docs=no-sphinx-pdfs --docs=no-sphinx-man", "RUNTEST_ARGS": "", @@ -6196,8 +6198,8 @@ "BUILD_FLAVOUR": "validate", "CONFIGURE_ARGS": "--with-intree-gmp --enable-strict-ghc-toolchain-check", "CONFIGURE_WRAPPER": "emconfigure", - "CROSS_STAGE": "2", "CROSS_TARGET": "javascript-unknown-ghcjs", + "FINAL_CROSS_STAGE": "2", "INSTALL_CONFIGURE_ARGS": "--enable-strict-ghc-toolchain-check", "RUNTEST_ARGS": "-e config.timeout=900", "TEST_ENV": "x86_64-linux-deb11-emsdk-closure-int_native-cross_javascript-unknown-ghcjs-validate", @@ -6439,7 +6441,7 @@ ".gitlab/ci.sh setup", ".gitlab/ci.sh configure", ".gitlab/ci.sh build_hadrian", - ".gitlab/ci.sh test_hadrian" + ".gitlab/ci.sh smoke_test" ], "stage": "full-build", "tags": [ @@ -6451,8 +6453,8 @@ "BUILD_FLAVOUR": "validate", "CONFIGURE_ARGS": "--with-intree-gmp --enable-strict-ghc-toolchain-check", "CROSS_EMULATOR": "qemu-aarch64 -L /usr/aarch64-linux-gnu", - "CROSS_STAGE": "2", "CROSS_TARGET": "aarch64-linux-gnu", + "FINAL_CROSS_STAGE": "2", "INSTALL_CONFIGURE_ARGS": "--enable-strict-ghc-toolchain-check", "RUNTEST_ARGS": "-e config.timeout=900", "TEST_ENV": "x86_64-linux-deb13-cross_aarch64-linux-gnu-validate", @@ -6726,7 +6728,8 @@ "paths": [ "ghc-x86_64-linux-deb13-riscv-cross_riscv64-linux-gnu-validate.tar.xz", "junit.xml", - "unexpected-test-output.tar.gz" + "unexpected-test-output.tar.gz", + "ghc-riscv64-linux-deb13-validate.tar.xz" ], "reports": { "junit": "junit.xml" @@ -6759,7 +6762,7 @@ ".gitlab/ci.sh setup", ".gitlab/ci.sh configure", ".gitlab/ci.sh build_hadrian", - ".gitlab/ci.sh test_hadrian" + ".gitlab/ci.sh smoke_test" ], "stage": "full-build", "tags": [ @@ -6768,11 +6771,12 @@ "variables": { "BIGNUM_BACKEND": "gmp", "BIN_DIST_NAME": "ghc-x86_64-linux-deb13-riscv-cross_riscv64-linux-gnu-validate", + "BIN_DIST_NAME_STAGE3": "ghc-riscv64-linux-deb13-validate", "BUILD_FLAVOUR": "validate", "CONFIGURE_ARGS": "--with-intree-gmp --enable-strict-ghc-toolchain-check", "CROSS_EMULATOR": "qemu-riscv64 -L /usr/riscv64-linux-gnu", - "CROSS_STAGE": "2", "CROSS_TARGET": "riscv64-linux-gnu", + "FINAL_CROSS_STAGE": "3", "INSTALL_CONFIGURE_ARGS": "--enable-strict-ghc-toolchain-check", "RUNTEST_ARGS": "-e config.timeout=900", "TEST_ENV": "x86_64-linux-deb13-riscv-cross_riscv64-linux-gnu-validate", @@ -7650,7 +7654,7 @@ ".gitlab/ci.sh setup", ".gitlab/ci.sh configure", ".gitlab/ci.sh build_hadrian", - ".gitlab/ci.sh test_hadrian" + ".gitlab/ci.sh smoke_test" ], "stage": "full-build", "tags": [ @@ -7662,8 +7666,8 @@ "BUILD_FLAVOUR": "validate", "CONFIGURE_ARGS": "--with-intree-gmp --enable-strict-ghc-toolchain-check", "CROSS_EMULATOR": "qemu-loongarch64 -L /usr/loongarch64-linux-gnu", - "CROSS_STAGE": "2", "CROSS_TARGET": "loongarch64-linux-gnu", + "FINAL_CROSS_STAGE": "2", "INSTALL_CONFIGURE_ARGS": "--enable-strict-ghc-toolchain-check", "RUNTEST_ARGS": "-e config.timeout=900", "TEST_ENV": "x86_64-linux-ubuntu24_04-loongarch-cross_loongarch64-linux-gnu-validate", ===================================== changelog.d/stage3-cross-bindists ===================================== @@ -0,0 +1,6 @@ +section: packaging +synopsis: Fully cross-compiled binary distributions +mrs: !15417 +issues: #26924 +description: Hadrian can now build cross-compiled binary distributions that run + native on another target architecture (e.g. AArch64 -> RISC-V). ===================================== distrib/configure.ac.in ===================================== @@ -9,7 +9,7 @@ AC_INIT([The Glorious Glasgow Haskell Compilation System], [@ProjectVersion@], [ dnl See /configure.ac for rationale. AC_PREREQ([2.69]) -AC_CONFIG_MACRO_DIRS([../m4]) +AC_CONFIG_MACRO_DIRS([m4]) dnl-------------------------------------------------------------------- dnl * Deal with arguments telling us gmp is somewhere odd ===================================== hadrian/src/BindistConfig.hs ===================================== @@ -24,6 +24,10 @@ crossBindist = BindistConfig { library_stage = Stage2, executable_stage = Stage1 targetBindist :: BindistConfig targetBindist = BindistConfig { library_stage = Stage2, executable_stage = Stage2 } +-- | Folder relative to build root ("bindist" or "bindist-stage3") +bindistFolder :: BindistConfig -> FilePath +bindistFolder conf | executable_stage conf == Stage2 = "bindist-stage3" +bindistFolder _conf = "bindist" -- | The implicit bindist config, if we don't know any better. implicitBindistConfig :: Action BindistConfig ===================================== hadrian/src/Rules/BinaryDist.hs ===================================== @@ -13,66 +13,89 @@ import Rules.Generate (generateSettings) import Settings import qualified System.Directory.Extra as IO import Settings.Program (programContext) -import Target -import Utilities import BindistConfig {- Note [Binary distributions] ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Hadrian produces binary distributions under: +Hadrian produces binary distributions that run on the build host architecture +(build == host, target == host || target /= host) under: <build root>/bindist/ghc-<X>.<Y>.<Z>-<arch>-<os>.tar.xz -It is generated by creating an archive from: - <build root>/bindist/ghc-<X>.<Y>.<Z>-<arch>-<os>/ +For stage3 (build /= host, host == target) bindists, a separate parent folder +is used: + <build root>/bindist-stage3/ghc-<X>.<Y>.<Z>-<arch>-<os>.tar.xz -It does so by following the steps below. +While regular bindists are usual same-arch compilers or cross-compilers, stage3 +bindists are cross-compiled compilers. -- make sure we have a complete stage 2 compiler + haddock +Bindists are generated by creating an archive from: + <build root>/<bindist|bindist-stage3>/ghc-<X>.<Y>.<Z>-<arch>-<os>/ + +Stage2 cross-compilers are a by-product of creating stage3 cross-compiled +compilers. The additional build dir (bindist-stage3) lets us keep both, such +that we can build them in one go on CI. +Configuration files (e.g. configure script and default.host.target) differ in +this case and keeping both targets separated also saves us some headache +dealing with stale files. + +This table introduces variables to simplify the following step descriptions: + +| compiler kind | <bindist-dir> | <executable-stage-dir> | <library-stage-dir> | +|------------------|----------------|------------------------|---------------------| +| native | bindist | stage1/ | stage1/ | +| cross-compiler | bindist | stage1/ | stage2/ | +| cross-compiled | bindist-stage3 | stage2/ | stage2/ | + +These are the steps to build a bindist: + +- make sure we have a complete compiler + libraries + haddock for the stage(s) + to bundle - copy the specific binaries which should be in the bindist to the bin folder and add the version suffix: - <build root>/stage1/bin/xxxx + <build root>/<executable-stage-dir>/bin/xxxx to - <build root/bindist/ghc-<X>.<Y>.<Z>-<arch>-<os>/bin/xxxx-<VER> + <build root>/<bindist-dir>/ghc-<X>.<Y>.<Z>-<arch>-<os>/bin/[<target>-]xxxx-<VER> + where the optional <target>- prefix is the cross triple for cross-compilers. - create symlink (or bash) wrapper from unversioned to versioned executable: - <build root/bindist/ghc-<X>.<Y>.<Z>-<arch>-<os>/bin/xxxx + <build root>/<bindist-dir>/ghc-<X>.<Y>.<Z>-<arch>-<os>/bin/[<target>-]xxxx points to: - <build root/bindist/ghc-<X>.<Y>.<Z>-<arch>-<os>/bin/xxxx-<VER> + <build root>/<bindist-dir>/ghc-<X>.<Y>.<Z>-<arch>-<os>/bin/[<target>-]xxxx-<VER> - copy the lib directories of the compiler we built: - <build root>/stage1/lib + <build root>/<library-stage-dir>/lib to - <build root>/bindist/ghc-<X>.<Y>.<Z>-<arch>-<os>/lib + <build root>/<bindist-dir>/ghc-<X>.<Y>.<Z>-<arch>-<os>/lib - copy the generated docs (user guide, haddocks, etc): - <build root>/docs/ + <build root>/doc/ to - <build root>/bindist/ghc-<X>.<Y>.<Z>-<arch>-<os>/docs/ + <build root>/<bindist-dir>/ghc-<X>.<Y>.<Z>-<arch>-<os>/doc/ -- use autoreconf to generate a `configure` script from - aclocal.m4 and distrib/configure.ac, that we move to: - <build root>/bindist/ghc-<X>.<Y>.<Z>-<arch>-<os>/configure +- use autoreconf to generate a staged `configure` script in + <build root>/<executable-stage>/distrib + that we move to: + <build root>/<bindist-dir>/ghc-<X>.<Y>.<Z>-<arch>-<os>/configure - write a (fixed) Makefile capable of supporting 'make install' to: - <build root>/bindist/ghc-<X>.<Y>.<Z>-<arch>-<os>/Makefile + <build root>/<bindist-dir>/ghc-<X>.<Y>.<Z>-<arch>-<os>/Makefile - write some (fixed) supporting bash code for the wrapper scripts to: - <build root>/bindist/ghc-<X>.<Y>.<Z>-<arch>-<os>/wrappers/<program> + <build root>/<bindist-dir>/ghc-<X>.<Y>.<Z>-<arch>-<os>/wrappers/<program> where <program> is the name of the executable that the bash file will help wrapping. -- copy supporting configure/make related files - (see @bindistInstallFiles@) to: - <build root>/bindist/ghc-<X>.<Y>.<Z>-<arch>-<os>/<file> +- copy supporting configure/make related files (see @bindistInstallFiles@) to: + <build root>/<bindist-dir>/ghc-<X>.<Y>.<Z>-<arch>-<os>/<file> - create a .tar.xz archive of the directory: - <build root>/bindist/ghc-<X>.<Y>.<Z>-<arch>-<os>/ + <build root>/<bindist-dir>/ghc-<X>.<Y>.<Z>-<arch>-<os>/ at - <build root>/bindist/ghc-<X>.<Y>.<Z>-<arch>-<os>.tar.xz + <build root>/<bindist-dir>/ghc-<X>.<Y>.<Z>-<arch>-<os>.tar.xz Note [Wrapper scripts and binary distributions] @@ -159,12 +182,12 @@ buildBinDistDir root conf@BindistConfig{..} = do distDir <- Context.distDir (vanillaContext library_stage rts) let ghcBuildDir = root -/- stageString library_stage - bindistFilesDir = root -/- "bindist" -/- ghcVersionPretty + bindistFilesDir = root -/- bindistFolder conf -/- ghcVersionPretty ghcVersionPretty = "ghc-" ++ version ++ "-" ++ targetPlatform rtsIncludeDir = distDir -/- "include" - -- We create the bindist directory at <root>/bindist/ghc-X.Y.Z-platform/ - -- and populate it with Stage2 build results + -- We create the bindist directory at <root>/<bindistFolder>/ghc-X.Y.Z-platform/ + -- and populate it with build results createDirectory bindistFilesDir createDirectory (bindistFilesDir -/- "bin") createDirectory (bindistFilesDir -/- "lib") @@ -230,9 +253,9 @@ buildBinDistDir root conf@BindistConfig{..} = do -- relocatable. The package DB is always at "package.conf.d" relative to -- the lib dir, matching the known bindist layout. let bindistSettings = bindistFilesDir -/- "lib" -/- "settings" - bindistContext = vanillaContext library_stage compiler + bindistContext = vanillaContext executable_stage compiler bindistSettingsContent <- interpretInContext bindistContext $ - generateSettings bindistSettings False "package.conf.d" + generateSettings bindistSettings False "package.conf.d" library_stage writeFileAtomic bindistSettings bindistSettingsContent copyDirectory rtsIncludeDir bindistFilesDir @@ -244,10 +267,10 @@ buildBinDistDir root conf@BindistConfig{..} = do -- -- N.B. the ghc-pkg executable may be prefixed with a target triple -- (c.f. #20267). - - -- Not going to work for cross - ghcPkgName <- programName (vanillaContext Stage1 ghcPkg) - ghcPkgProg <- exeSpawnPath (bindistFilesDir -/- "bin" -/- ghcPkgName) + -- Stage1 is also correct for cross-compiler scenarios, because it runs on + -- the build host. + ghcPkgPath <- programPath =<< programContext Stage1 ghcPkg + ghcPkgProg <- exeSpawnPath ghcPkgPath cmdExe ghcPkgProg ["recache", "--package-db", bindistFilesDir -/- "lib" -/- "package.conf.d" ] :: Action () @@ -345,8 +368,8 @@ bindistRules = do buildBinDistDir root cfg phony "binary-dist-dir-cross" $ buildBinDistDir root crossBindist - -- MP: Not working yet - -- phony "binary-dist-dir-stage3" $ buildBinDistDir root targetBindist + + phony "binary-dist-dir-stage3" $ buildBinDistDir root targetBindist let buildBinDist compressor = do win_host <- isWinHost @@ -378,49 +401,27 @@ bindistRules = do phony (name <> "-dist-xz") $ mk_bindist Xz phony "binary-dist-cross" $ buildBinDistX "binary-dist-dir-cross" "bindist" Xz - phony "binary-dist-stage3" $ buildBinDistX "binary-dist-dir-stage3" "bindist" Xz - - -- Prepare binary distribution configure script - -- (generated under <ghc root>/distrib/configure by 'autoreconf') - root -/- "bindist" -/- "ghc-*" -/- "configure" %> \configurePath -> do - need ["distrib" -/- "configure.ac"] - ghcRoot <- topDirectory - copyFile (ghcRoot -/- "aclocal.m4") (ghcRoot -/- "distrib" -/- "aclocal.m4") - copyDirectory (ghcRoot -/- "m4") (ghcRoot -/- "distrib") - - -- Note [Autoreconf unix paths from ACLOCAL_PATH] - -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -- On Windows, autoreconf fails when the ACLOCAL_PATH env variable contains Windows- - -- style paths. This happens because MSYS2 automatically converts env variables to - -- Windows-style paths. To fix this, we convert ACLOCAL_PATH back to Unix style. - -- This is done both in the boot Python script and here when building a bindist. - -- See Note [MSYS paths]. - win_host <- isWinHost - aclocalPathMay <- if win_host then getEnv "ACLOCAL_PATH" else pure Nothing - let env = [ AddEnv "ACLOCAL_PATH" (windowsToMsysPathList aclocalPath) - | Just aclocalPath <- [aclocalPathMay] ] - - buildWithCmdOptions env $ - target (vanillaContext Stage1 ghc) (Autoreconf $ ghcRoot -/- "distrib") [] [] - -- We clean after ourselves, moving the configure script we generated in - -- our bindist dir - removeFile (ghcRoot -/- "distrib" -/- "aclocal.m4") - removeDirectory (ghcRoot -/- "distrib" -/- "m4") - - moveFile (ghcRoot -/- "distrib" -/- "configure") configurePath - - -- Generate the Makefile that enables the "make install" part - root -/- "bindist" -/- "ghc-*" -/- "Makefile" %> \makefilePath -> do - top <- topDirectory - copyFile (top -/- "hadrian" -/- "bindist" -/- "Makefile") makefilePath - - -- Copy various configure-related files needed for a working - -- './configure [...] && make install' workflow - -- (see the list of files needed in the 'binary-dist' rule above, before - -- creating the archive). - forM_ bindistInstallFiles $ \file -> - root -/- "bindist" -/- "ghc-*" -/- file %> \dest -> do - copyFile (fixup file) dest + phony "binary-dist-stage3" $ buildBinDistX "binary-dist-dir-stage3" "bindist-stage3" Xz + + forM_ [normalBindist, targetBindist] $ \bindistCfg -> do + let bindistFolderName = bindistFolder bindistCfg + root -/- bindistFolderName -/- "ghc-*" -/- "configure" %> \configurePath -> do + let distribConfigure = root -/- stageString (executable_stage bindistCfg) -/- "distrib" -/- "configure" + need [distribConfigure] + copyFile distribConfigure configurePath + + -- Copy the Makefile that enables the "make install" part + root -/- bindistFolderName -/- "ghc-*" -/- "Makefile" %> \makefilePath -> do + top <- topDirectory + copyFile (top -/- "hadrian" -/- "bindist" -/- "Makefile") makefilePath + + -- Copy various configure-related files needed for a working + -- './configure [...] && make install' workflow + -- (see the list of files needed in the 'binary-dist' rule above, before + -- creating the archive). + forM_ bindistInstallFiles $ \file -> + root -/- bindistFolderName -/- "ghc-*" -/- file %> \dest -> do + copyFile (fixup file) dest where fixup f | f `elem` ["INSTALL", "README"] = "distrib" -/- f @@ -500,7 +501,7 @@ pkgToWrappers stage pkg = do -- These are the packages which we want to expose to the user and hence -- there are wrappers installed in the bindist. | pkg `elem` [hpcBin, haddock, hp2ps, hsc2hs, ghc, ghcPkg] - -> (:[]) <$> (programName =<< programContext Stage1 pkg) + -> (:[]) <$> (programName =<< programContext stage pkg) | otherwise -> pure [] wrapper :: Stage -> FilePath -> Action String @@ -553,7 +554,7 @@ ghciScriptWrapper stage = do -- | Create a wrapper script calls the executable given as first argument createVersionWrapper :: Stage -> Package -> String -> FilePath -> Action () createVersionWrapper executable_stage pkg versioned_exe install_path = do - ghcPath <- builderPath (Ghc CompileCWithGhc (succStage executable_stage)) + ghcPath <- builderPath (Ghc CompileCWithGhc executable_stage) top <- topDirectory let version_wrapper_dir = top -/- "hadrian" -/- "bindist" -/- "cwrappers" wrapper_files = [ version_wrapper_dir -/- file | file <- ["version-wrapper.c", "getLocation.c", "cwrapper.c"]] ===================================== hadrian/src/Rules/Generate.hs ===================================== @@ -24,7 +24,6 @@ import Utilities import GHC.Toolchain as Toolchain hiding (HsCpp(HsCpp)) import GHC.Platform.ArchOS import Settings.Program (ghcWithInterpreter) -import UserSettings (finalStage) -- | Track this file to rebuild generated files whenever it changes. trackGenerateHs :: Expr () @@ -251,32 +250,33 @@ generateRules = do (root -/- "ghc-stage2") <~+ ghcWrapper Stage2 (root -/- "ghc-stage3") <~+ ghcWrapper Stage3 - forM_ allStages $ \stage -> do - let prefix = root -/- stageString stage -/- "lib" - -- For the finalStage, we generate settings for that stage. For - -- others we look at the next stage. Why? Because cross-compilers - -- require libs from the successor stage, otherwise they are - -- compiled for the host and not the target. - stage' = if stage /= finalStage then succStage stage else stage - go gen file = generate file (semiEmptyTarget stage') gen + forM_ allStages $ \compilerStage -> do + let + prefix = root -/- stageString compilerStage -/- "lib" + go gen file = generate file (semiEmptyTarget compilerStage) gen (prefix -/- "settings") %> \out -> do - let get_pkg_db stg = packageDbPath (PackageDbLoc stg Final) - -- For cross, LibDir points to stage' lib dir, so pkgDb must also - -- be relative to stage' lib dir. - isCross <- crossStage stage - let libStage = case stage of + -- @compilerStage@ is the stage that builds the compiler, i.e. the + -- binary whose settings these are; @libraryStage@ is the stage + -- whose libraries and package DB that compiler uses. The two + -- differ in two cases: Stage0 has no library or package DB of its + -- own (the bootstrapping compiler uses Stage1's), and cross + -- stages use the successor stage's libraries (a cross compiler + -- requires libs from the successor stage, otherwise they are + -- compiled for the host and not the target). + isCross <- crossStage compilerStage + let libraryStage = case compilerStage of Stage0 {} -> Stage1 - _ -> if isCross then stage' else stage - pkgDb <- get_pkg_db libStage + _ -> if isCross then succStage compilerStage else compilerStage + pkgDb <- packageDbPath (PackageDbLoc libraryStage Final) -- addTrailingPathSeparator needed: makeRelativeNoSysLink uses -- splitPath where "lib" and "lib/" are distinct components. let libTopDir = addTrailingPathSeparator $ - if isCross - then root -/- stageString stage' -/- "lib" - else prefix + if isStage0 compilerStage + then prefix + else root -/- stageString libraryStage -/- "lib" relPkgDb = makeRelativeNoSysLink libTopDir pkgDb - go (generateSettings out True relPkgDb) out - (prefix -/- "targets" -/- "default.target") %> \out -> go (show <$> expr (targetStage (succStage stage))) out + go (generateSettings out True relPkgDb libraryStage) out + (prefix -/- "targets" -/- "default.target") %> \out -> go (show <$> expr (targetStage (succStage compilerStage))) out where file <~+ gen = file %> \out -> generate out emptyTarget gen >> makeExecutable out @@ -412,6 +412,7 @@ templateRules = do bindistRules :: Rules () bindistRules = do + root <- buildRootRules templateRule ("mk" -/- "project.mk") $ mconcat [ interpolateSetting "ProjectName" ProjectName , interpolateSetting "ProjectVersion" ProjectVersion @@ -423,43 +424,124 @@ bindistRules = do , interpolateVar "HostOS_CPP" $ fmap cppify $ interp $ queryHost queryOS - , interpolateVar "TargetPlatform" $ getTarget targetPlatformTriple - , interpolateVar "TargetPlatform_CPP" $ cppify <$> getTarget targetPlatformTriple - , interpolateVar "TargetArch_CPP" $ cppify <$> getTarget queryArch - , interpolateVar "TargetOS_CPP" $ cppify <$> getTarget queryOS - , interpolateVar "LLVMTarget" $ getTarget tgtLlvmTarget - ] - templateRule ("distrib" -/- "configure.ac") $ mconcat - [ interpolateSetting "ConfiguredEmsdkVersion" EmsdkVersion - , interpolateVar "CrossCompilePrefix" $ do - crossCompiling <- interp $ getFlag CrossCompiling - tpf <- setting TargetPlatformFull - pure $ if crossCompiling then tpf <> "-" else "" - , interpolateVar "LeadingUnderscore" $ yesNo <$> getTarget tgtSymbolsHaveLeadingUnderscore - , interpolateSetting "LlvmMaxVersion" LlvmMaxVersion - , interpolateSetting "LlvmMinVersion" LlvmMinVersion - , interpolateVar "LlvmTarget" $ getTarget tgtLlvmTarget - , interpolateSetting "ProjectVersion" ProjectVersion - , interpolateVar "EnableDistroToolchain" $ interp (staged (lookupStageBuildConfig "settings-use-distro-mingw")) - , interpolateVar "TablesNextToCode" $ yesNo <$> getTarget tgtTablesNextToCode - , interpolateVar "TargetHasLibm" $ yesNo <$> getTarget tgtHasLibm - , interpolateVar "TargetPlatform" $ getTarget targetPlatformTriple - , interpolateVar "BuildPlatform" $ interp $ queryBuild targetPlatformTriple - , interpolateVar "HostPlatform" $ interp $ queryHost targetPlatformTriple - , interpolateVar "TargetWordBigEndian" $ getTarget isBigEndian - , interpolateVar "TargetWordSize" $ getTarget wordSize - , interpolateVar "Unregisterised" $ yesNo <$> getTarget tgtUnregisterised - , interpolateVar "UseLibdw" $ fmap yesNo $ interp $ staged (fmap (isJust . tgtRTSWithLibdw) . targetStage) - , interpolateVar "UseLibffiForAdjustors" $ yesNo <$> getTarget tgtUseLibffiForAdjustors - , interpolateVar "BaseUnitId" $ pkgUnitId Stage1 base - , interpolateVar "GhcWithSMP" $ yesNo <$> targetSupportsSMP Stage2 - , interpolateVar "TargetPlatformFull" (setting TargetPlatformFull) - , interpolateVar "BuildPlatformFull" (setting BuildPlatformFull) - , interpolateVar "HostPlatformFull" (setting HostPlatformFull) + -- Stage2 always targets the target (as in build, host, target) + -- architecture. Thus, we can use it constant stage here to find + -- information about the target architecture. + , interpolateVar "TargetPlatform" $ getTarget Stage2 targetPlatformTriple + , interpolateVar "TargetPlatform_CPP" $ cppify <$> getTarget Stage2 targetPlatformTriple + , interpolateVar "TargetArch_CPP" $ cppify <$> getTarget Stage2 queryArch + , interpolateVar "TargetOS_CPP" $ cppify <$> getTarget Stage2 queryOS + , interpolateVar "LLVMTarget" $ getTarget Stage2 tgtLlvmTarget ] + forM_ [Stage1, Stage2] $ \stage -> + let crossStageInterps = Interpolations $ do + isCrossStage <- crossStage stage + targetPlatform <- setting TargetPlatformFull + -- For cross-compiled compilers we need to pretend that they were + -- build on the target. For regular compilers we can assume that: + -- build == host == target + buildPlatform <- + if isCrossStage + then + interp $ queryBuild targetPlatformTriple + else getTarget stage targetPlatformTriple + hostPlatform <- + if isCrossStage + then + interp $ queryHost targetPlatformTriple + else getTarget stage targetPlatformTriple + baseUnitId <- pkgUnitId (if isCrossStage then succStage stage else stage) base + buildPlatformFull <- if isCrossStage then setting BuildPlatformFull else setting TargetPlatformFull + hostPlatformFull <- if isCrossStage then setting HostPlatformFull else setting TargetPlatformFull + pure + [ ("CrossCompilePrefix", if isCrossStage then targetPlatform <> "-" else "") + , ("TargetPlatformFull", targetPlatform) + , ("BuildPlatform", buildPlatform) + , ("HostPlatform", hostPlatform) + , ("BaseUnitId", baseUnitId) + , ("BuildPlatformFull", buildPlatformFull) + , ("HostPlatformFull", hostPlatformFull) + ] + in templateRuleFrom + ("distrib" -/- "configure.ac" <.> "in") + (root -/- stageString stage -/- "distrib" -/- "configure.ac") + $ mconcat + [ interpolateSetting "ConfiguredEmsdkVersion" EmsdkVersion + , interpolateVar "LeadingUnderscore" $ yesNo <$> getLibTarget stage tgtSymbolsHaveLeadingUnderscore + , interpolateSetting "LlvmMaxVersion" LlvmMaxVersion + , interpolateSetting "LlvmMinVersion" LlvmMinVersion + , interpolateVar "LlvmTarget" $ getLibTarget stage tgtLlvmTarget + , interpolateSetting "ProjectVersion" ProjectVersion + , interpolateVar "EnableDistroToolchain" $ interp (staged (lookupStageBuildConfig "settings-use-distro-mingw")) + , interpolateVar "TablesNextToCode" $ yesNo <$> getLibTarget stage tgtTablesNextToCode + , interpolateVar "TargetHasLibm" $ yesNo <$> getLibTarget stage tgtHasLibm + , interpolateVar "TargetPlatform" $ getLibTarget stage targetPlatformTriple + , interpolateVar "TargetWordBigEndian" $ getLibTarget stage isBigEndian + , interpolateVar "TargetWordSize" $ getLibTarget stage wordSize + , interpolateVar "Unregisterised" $ yesNo <$> getLibTarget stage tgtUnregisterised + , interpolateVar "UseLibdw" $ yesNo <$> getLibTarget stage (isJust . tgtRTSWithLibdw) + , interpolateVar "UseLibffiForAdjustors" $ yesNo <$> getLibTarget stage tgtUseLibffiForAdjustors + , interpolateVar "GhcWithSMP" $ yesNo <$> libStageSupportsSMP stage + , crossStageInterps + ] + + -- We can build two kinds of bindists: Regular Stage2 (including + -- cross-compilers) and fully cross-compiled Stage3. To avoid + -- race-conditions, stale files, etc. build the `configure` scripts as part + -- of the stage's _build files. This requires copying several files such that + -- they are available to the autoconf run. + forM_ [Stage1, Stage2] $ \stage -> do + let distribDir = root -/- stageString stage -/- "distrib" + + distribDir -/- "aclocal.m4" %> \out -> do + top <- topDirectory + copyFile (top -/- "aclocal.m4") out + + forM_ ["config.sub", "config.guess", "install-sh"] $ \f -> + distribDir -/- f %> \out -> do + top <- topDirectory + copyFile (top -/- f) out + + distribDir -/- "m4/*.m4" %> \out -> do + top <- topDirectory + copyFile (top -/- "m4" -/- takeFileName out) out + + distribDir -/- "configure" %> \_ -> do + top <- topDirectory + m4Files <- getDirectoryFiles (top -/- "m4") ["*.m4"] + need $ [ distribDir -/- "configure.ac" + , distribDir -/- "config.sub" + , distribDir -/- "config.guess" + , distribDir -/- "install-sh" + , distribDir -/- "aclocal.m4" + ] + ++ [ distribDir -/- "m4" -/- takeFileName f | f <- m4Files ] + + -- Note [Autoreconf unix paths from ACLOCAL_PATH] + -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + -- On Windows, autoreconf fails when the ACLOCAL_PATH env variable + -- contains Windows-style paths. MSYS2 auto-converts env vars to + -- Windows-style, so we convert ACLOCAL_PATH back to Unix style here. + -- This is done both in the boot Python script and here when + -- building a bindist. + -- See Note [MSYS paths]. + win_host <- isWinHost + aclocalPathMay <- if win_host then getEnv "ACLOCAL_PATH" else pure Nothing + let env = [ AddEnv "ACLOCAL_PATH" (windowsToMsysPathList aclocalPath) + | Just aclocalPath <- [aclocalPathMay] ] + + buildWithCmdOptions env $ + target (vanillaContext stage ghc) (Autoreconf distribDir) [] [] where interp = interpretInContext (semiEmptyTarget Stage2) - getTarget = interp . queryTarget Stage2 + getTarget stage = interp . queryTarget stage + getLibTarget executableStage f = do + isCross <- crossStage executableStage + getTarget (if isCross then succStage executableStage else executableStage) f + -- | 'targetSupportsSMP' lifted to the library stage (see 'getLibTarget'). + libStageSupportsSMP stage = do + isCross <- crossStage stage + targetSupportsSMP (if isCross then succStage stage else stage) -- | Given a 'String' replace characters '.' and '-' by underscores ('_') so that -- the resulting 'String' is a valid C preprocessor identifier. @@ -481,42 +563,37 @@ ghcWrapper stage = do -- | Generate settings file, optionally including @LibDir@. -- +-- Describes the compiler whose stage is the ambient 'Expr' context +-- (available here as @executableStage@ via 'getStage'). The @libraryStage@ +-- argument is the stage whose lib dir holds the libraries the described +-- compiler links against — used both for the @base@ unit-id lookup and for +-- the @LibDir@ entry. It usually equals @executableStage@ but differs when +-- the compiler links against libraries from a different stage (cross +-- compilers, or the Stage0 bootstrap compiler using Stage1's libraries). +-- -- @rel_pkg_db@: package DB path relative to the lib dir (e.g. -- "package.conf.d"). Callers supply the correct relative path. For bindists --- the layout is known statically; for in-tree builds callers compute it. For --- bindists, we omit @LibDir@ so it defaults to @topDir@ at runtime. -generateSettings :: FilePath -> Bool -> FilePath -> Expr String -generateSettings settingsFile includeLibDir rel_pkg_db = do +-- the layout is known statically; for in-tree builds callers compute it. +-- For bindists, we omit @LibDir@ so it defaults to @topDir@ at runtime. +generateSettings :: FilePath -> Bool -> FilePath -> Stage -> Expr String +generateSettings settingsFile includeLibDir rel_pkg_db libraryStage = do ctx <- getContext - stage <- getStage + executableStage <- getStage + + base_unit_id <- expr $ pkgUnitId libraryStage base - -- The unit-id of the base package which is always linked against (#25382) - base_unit_id <- expr $ do - case stage of - Stage0 {} -> error "Unable to generate settings for stage0" - Stage1 -> pkgUnitId Stage1 base - Stage2 -> pkgUnitId Stage1 base - Stage3 -> pkgUnitId Stage2 base - - -- For cross compilers, LibDir points to the succeeding stage's lib dir - -- (which contains the target architecture's libraries). For non-cross, - -- it points to the preceding stage's lib dir as usual. - let compilerStage = predStage stage -- the GHC that builds packages in this stage - isCrossLibDir <- expr $ crossStage compilerStage - let stage_dir_stage = if isCrossLibDir then stage else compilerStage - - -- addTrailingPathSeparator is needed because makeRelativeNoSysLink uses - -- splitPath internally, where "lib" and "lib/" are distinct components. - lib_topDir :: FilePath <- expr $ addTrailingPathSeparator <$> stageLibPath stage_dir_stage + lib_topDir :: FilePath <- expr $ addTrailingPathSeparator <$> stageLibPath libraryStage let rel_lib_topDir = makeRelativeNoSysLink (dropFileName settingsFile) lib_topDir settings <- traverse sequence $ - [ ("unlit command", ("$topdir/../bin/" <>) <$> expr (programName (ctx { Context.package = unlit, Context.stage = compilerStage }))) - , ("Use interpreter", expr $ yesNo <$> ghcWithInterpreter compilerStage) - -- Hard-coded as Cabal queries these to determine way support and we - -- need to always advertise all ways when bootstrapping. - -- The settings file is generated at install time when installing a bindist. - , ("RTS ways", unwords . map show . Set.toList <$> getRtsWays) + [ ("unlit command", ("$topdir/../bin/" <>) <$> expr (programName (ctx { Context.package = unlit }))) + , ("Use interpreter", expr $ yesNo <$> ghcWithInterpreter executableStage) + -- Advertise the RTS ways that will actually ship with the compiler + -- described by this settings file, i.e. the ways the @libraryStage@ + -- RTS is built with. + -- The settings file is regenerated at install time when installing a bindist. + , ("RTS ways", unwords . map show . Set.toList <$> + expr (interpretInContext (vanillaContext libraryStage rts) getRtsWays)) , ("Relative Global Package DB", pure rel_pkg_db) , ("base unit-id", pure base_unit_id) ] View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/265b473308767b8299ba5cbf70b8138... -- View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/265b473308767b8299ba5cbf70b8138... You're receiving this email because of your account on gitlab.haskell.org. Manage all notifications: https://gitlab.haskell.org/-/profile/notifications | Help: https://gitlab.haskell.org/help