Simon Jakobi pushed to branch wip/sjakobi/T18922-filecheck-llvm-poc at Glasgow Haskell Compiler / GHC Commits: 45242359 by Simon Jakobi at 2026-11-05T06:25:13+01:00 testsuite: Extend the FileCheck package table to every GHC platform The first version of the table only covered the systems I had looked at. Checked on 2026-11-05 against package indexes, spec files and port recipes for each platform on the GHC platforms wiki page, which adds Rocky, Alpine, the archived Debian and Ubuntu releases, DragonFly and the architecture coverage. Also list the LLVM sources that do not ship FileCheck at all. The official release tarballs were previously marked as unchecked. Streaming the listing of the 15.0.6 tarball that the Rocky CI image uses showed no FileCheck among its 148 bin/ entries. Assisted-by: Claude Fable 5.1 - - - - - ceb5d2cc by Simon Jakobi at 2026-11-05T07:43:53+01:00 testsuite: Find FileCheck in configure instead of the driver The driver searched PATH itself for FileCheck under the Debian and FreeBSD names, with a hard-coded version range, and had grown probes for Alpine's and Homebrew's install prefixes. Configure already does this search for llc and opt, with the same name patterns and version parsing, so do the same for FileCheck and hand the result to the driver as config.filecheck. Users on platforms where the binary is off PATH pass FILECHECK= to configure, as they do for LLC=. The driver keeps the FILECHECK environment variable and the output of build_filecheck.py as overrides, since the latter runs after configure. Assisted-by: Claude Fable 5.1 - - - - - 81948d02 by Simon Jakobi at 2026-11-05T07:43:53+01:00 testsuite: Pass configure's FileCheck through the make runner too The make runner takes everything it knows about the compiler from ghc --info and never looks at configure's output, so after the previous commit it passed no config.filecheck and FileCheck tests were framework failures unless FILECHECK was set by hand. FileCheck is a testsuite tool rather than a compiler setting, so ghc --info cannot carry it. Read the one line from hadrian/cfg/system.config instead. In an unconfigured tree the value is empty and the driver falls back to the environment variable and the built binary as before. Assisted-by: Claude Fable 5.1 - - - - - 8 changed files: - configure.ac - hadrian/cfg/system.config.in - hadrian/src/Oracles/Setting.hs - hadrian/src/Settings/Builders/RunTest.hs - testsuite/driver/testglobals.py - testsuite/driver/testlib.py - testsuite/filecheck.md - testsuite/mk/test.mk Changes: ===================================== configure.ac ===================================== @@ -583,6 +583,20 @@ FIND_LLVM_PROG([LLVMAS], [clang], [$LlvmMinVersion], [$LlvmMaxVersion]) LlvmAsCmd="$LLVMAS" AC_SUBST([LlvmAsCmd]) +dnl ** Which LLVM FileCheck to use? (testsuite only) +dnl -------------------------------------------------------------- +# CQ[filecheck-version-range] +# Q: FileCheck is independent of the LLVM backend, so why the backend's +# LlvmMaxVersion as the upper bound, and what fixes the floor at 12? +# A~ FIND_LLVM_PROG needs a finite range to enumerate the suffixed names +# (FileCheck-NN, FileCheckNN). The testsuite passes +# --allow-unused-prefixes, which appeared in LLVM 12. +FileCheckMinVersion=12 # inclusive +AC_ARG_VAR(FILECHECK,[Use as the path to LLVM's FileCheck, for the testsuite [default=autodetect]]) +FIND_LLVM_PROG([FILECHECK], [FileCheck], [$FileCheckMinVersion], [$LlvmMaxVersion]) +FileCheckCmd="$FILECHECK" +AC_SUBST([FileCheckCmd]) + dnl -------------------------------------------------------------- dnl End of configure script option section dnl -------------------------------------------------------------- @@ -1077,7 +1091,8 @@ echo "\ llc : $LlcCmd opt : $OptCmd llvm-as : $LlvmAsCmd - llvm-dlltool : $DlltoolCmd" + llvm-dlltool : $DlltoolCmd + FileCheck : $FileCheckCmd" if test "$HSCOLOUR" = ""; then echo " ===================================== hadrian/cfg/system.config.in ===================================== @@ -23,6 +23,7 @@ makeinfo = @MAKEINFO@ bourne-shell = @SH@ git = @GIT@ cabal = @CABAL@ +filecheck = @FileCheckCmd@ # Python 3 is required to run test driver. # See: https://github.com/ghc/ghc/blob/master/testsuite/mk/boilerplate.mk#L220 ===================================== hadrian/src/Oracles/Setting.hs ===================================== @@ -62,6 +62,7 @@ data ProjectSetting = | HostPlatformFull | BourneShell | EmsdkVersion + | FileCheck -- | Things which configure how a specific stage is built data BuildSetting = @@ -105,6 +106,7 @@ setting key = lookupSystemConfig $ case key of HostPlatformFull -> "host-platform-full" BourneShell -> "bourne-shell" EmsdkVersion -> "emsdk-version" + FileCheck -> "filecheck" buildSetting :: BuildSetting -> Stage -> Action String buildSetting key stage = tgtConfig stage $ case key of ===================================== hadrian/src/Settings/Builders/RunTest.hs ===================================== @@ -238,6 +238,7 @@ runTestBuilderArgs = builder Testsuite ? do (testEnv, testMetricsFile) <- expr . liftIO $ (,) <$> lookupEnv "TEST_ENV" <*> lookupEnv "METRICS_FILE" perfBaseline <- expr . liftIO $ lookupEnv "PERF_BASELINE_COMMIT" + filecheck <- expr $ setting FileCheck targetWrapper <- expr . liftIO $ lookupEnv "CROSS_EMULATOR" threads <- shakeThreads <$> expr getShakeOptions @@ -282,6 +283,7 @@ runTestBuilderArgs = builder Testsuite ? do , arg "-e", arg $ "config.package_conf_cache_file=" ++ show pkgConfCacheFile , arg "-e", arg $ "config.libdir=" ++ show libdir + , arg "-e", arg $ "config.filecheck=" ++ show filecheck , arg "-e", arg $ "config.have_interp=" ++ show withInterpreter ===================================== testsuite/driver/testglobals.py ===================================== @@ -129,9 +129,9 @@ class TestConfig: # Do we have a functional LLVM toolchain? self.have_llvm = False - # LLVM's FileCheck, for filecheck_stderr and filecheck_stdout - # (None: look for it when first needed, see find_filecheck) - self.filecheck = None # type: Optional[str] + # LLVM's FileCheck as found by configure, for filecheck_stderr and + # filecheck_stdout (empty: none found, see find_filecheck) + self.filecheck = '' # Do we have vanilla libraries? self.have_vanilla = False ===================================== testsuite/driver/testlib.py ===================================== @@ -1295,30 +1295,18 @@ def _filecheck(name, opts, kind, check_file, prefixes): # have_llvm() tests merely skip; is loud right for a tool only some # tests need? if find_filecheck() is None: - framework_fail(name, None, 'FileCheck not found: set FILECHECK or ' - 'config.filecheck, or run testsuite/driver/build_filecheck.py') + framework_fail(name, None, 'FileCheck not found: pass FILECHECK=... to ' + 'configure, or run testsuite/driver/build_filecheck.py') opts.skip = True opts.filecheck[kind] = (name, check_file, prefixes or ['CHECK']) -# CQ[filecheck-discovery] -# Q: should configure find FileCheck like llc (FIND_LLVM_PROG) and hadrian -# pass it as config.filecheck, instead of this search from the driver? -# CQ[filecheck-version-range] -# Q: the versioned names mirror configure's LlvmMinVersion..LlvmMaxVersion; -# which FileCheck versions do the tests actually require? def find_filecheck() -> Optional[str]: - if config.filecheck is None: - built = config.top.parent / '_build' / 'filecheck' / 'bin' / 'FileCheck' - candidates = [os.environ.get('FILECHECK'), str(built), 'FileCheck'] \ - + ['FileCheck-%d' % v for v in range(23, 12, -1)] - for c in candidates: - found = shutil.which(c) if c else None - if found: - config.filecheck = found - break - else: - config.filecheck = '' - return config.filecheck or None + built = config.top.parent / '_build' / 'filecheck' / 'bin' / 'FileCheck' + for c in [os.environ.get('FILECHECK'), str(built), config.filecheck]: + found = shutil.which(c) if c else None + if found: + return found + return None def no_check_hp(name, opts): opts.check_hp = False ===================================== testsuite/filecheck.md ===================================== @@ -17,7 +17,7 @@ Installing FileCheck Without a `FileCheck` binary every test that uses it is reported as a framework failure and skipped: - FileCheck not found: set FILECHECK or config.filecheck, or run + FileCheck not found: pass FILECHECK=... to configure, or run testsuite/driver/build_filecheck.py Any LLVM 12 or newer should do. The directives and the regex dialect have not @@ -30,32 +30,49 @@ supported LLVM version range. In this order, taking the first that exists: -1. `config.filecheck`, which nothing in configure or hadrian sets yet. It can - be set with `-e config.filecheck=...` when invoking `runtests.py` - directly. -2. The `FILECHECK` environment variable, a path or a command name. -3. `_build/filecheck/bin/FileCheck` in the GHC tree, where +1. The `FILECHECK` environment variable at test time, a path or a command + name. +2. `_build/filecheck/bin/FileCheck` in the GHC tree, where `build_filecheck.py` installs to. -4. `FileCheck` on `PATH`, then `FileCheck-23` down to `FileCheck-13`, the - names Debian-style packages use. +3. What `./configure` found, passed as `config.filecheck` by hadrian and by + `make -C testsuite/tests`, which reads it from `hadrian/cfg/system.config`. + Configure searches `PATH` the way it does for `llc`: `FileCheck-23` down + to `FileCheck-12` (the Debian and Ubuntu names), `FileCheck23` down to + `FileCheck12` (the FreeBSD names), then plain `FileCheck`, and rejects a + version outside that range. `./configure FILECHECK=/path/to/FileCheck` + overrides the search, which is needed where the binary is not on `PATH` + (Alpine, Homebrew). The result is in configure's summary under "Using + LLVM tools". ### Distribution packages Checked against the package file lists or build recipes on 2026-11-05, not by -installing each one. +installing each one. Versions below LLVM 12 do not work (see above). -| System | Package | Binary | Found automatically? | +| System | Package | Binary | Found by configure? | | --- | --- | --- | --- | -| Debian, Ubuntu | `llvm-NN-tools` | `/usr/bin/FileCheck-NN` | yes | +| Debian 11+, Ubuntu 20.04+ | `llvm-NN-tools` | `/usr/bin/FileCheck-NN` | yes | +| Debian 9, 10, Ubuntu 18.04 | `llvm-NN-tools` from [apt.llvm.org](https://apt.llvm.org) | `/usr/bin/FileCheck-NN` | yes | | Fedora | `llvm` | `/usr/bin/FileCheck` | yes | +| Rocky 8, 9 | `llvm` | `/usr/bin/FileCheck` | yes | +| Alpine 3.16+ | `llvmNN-test-utils` | `/usr/lib/llvmNN/bin/FileCheck` | no, pass `FILECHECK=` | | Arch | `llvm` | `/usr/bin/FileCheck` | yes | -| Homebrew | `llvm` (keg-only) | `$(brew --prefix llvm)/bin/FileCheck` | no, set `FILECHECK` | +| Homebrew | `llvm` or `llvm@NN` (keg-only) | `$(brew --prefix llvm)/bin/FileCheck` | no, pass `FILECHECK=` | | Nix | `llvmPackages.llvm` | `bin/FileCheck` of that package | yes, once in `PATH` | -| FreeBSD | `llvmNN` (`LIT` option, on by default) | `/usr/local/bin/FileCheckNN` | no, set `FILECHECK` | -| MSYS2 | `mingw-w64-<env>-llvm` | `FileCheck.exe` in the environment's `bin` | yes, in that shell | +| FreeBSD, DragonFly | `llvmNN` (`LIT` option, on by default) | `/usr/local/bin/FileCheckNN` | yes | +| MSYS2 (all environments, including CLANGARM64) | `mingw-w64-<env>-llvm` | `FileCheck.exe` in the environment's `bin` | yes, in that shell | -Not checked: the official `clang+llvm-*` release bindists (which GHC's CI -images install LLVM from) and LLVM's Windows installer. +The Debian and Ubuntu packages exist for amd64, i386, arm64, armhf, ppc64el, +s390x, riscv64 and loong64 (loong64 in unstable only). + +Sources that do **not** ship FileCheck, so use the build script below: + +- the official `clang+llvm-*` release tarballs and the LLVM Windows + installer, which are built without `LLVM_INSTALL_UTILS` +- llvm-mingw +- MacPorts `llvm-NN` +- OpenBSD's `devel/llvm`, which removes it on purpose +- Alpine 3.12 to 3.15, whose only `-test-utils` package is LLVM 10 The Debian package is the cheapest route: it is a few hundred kB and does not pull in the rest of LLVM. For example, on Ubuntu 24.04: ===================================== testsuite/mk/test.mk ===================================== @@ -284,6 +284,15 @@ RUNTEST_OPTS += --config 'stats_files_dir=$(TOP)/tests/perf/haddock' RUNTEST_OPTS += -e "config.stage=$(GhcStage)" +# CQ[filecheck-make-path] +# Q: Why read hadrian's system.config here, when nothing else in the make +# path depends on ./configure having run? +# A~ FileCheck is a testsuite tool, not a compiler setting, so ghc --info +# cannot supply it. An unconfigured tree yields an empty value and the +# driver falls back to $FILECHECK or _build/filecheck (see find_filecheck). +FILECHECK ?= $(shell sed -n 's/^filecheck *= *//p' '$(TOP)/../hadrian/cfg/system.config' 2>/dev/null) +RUNTEST_OPTS += -e 'config.filecheck=r"$(FILECHECK)"' + ifneq "$(METRICS_FILE)" "" RUNTEST_OPTS += \ --metrics-file "$(METRICS_FILE)" View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/ea277be8a411158a6ead78a22b37eff... -- View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/ea277be8a411158a6ead78a22b37eff... You're receiving this email because of your account on gitlab.haskell.org. Manage all notifications: https://gitlab.haskell.org/-/profile/notifications | Help: https://gitlab.haskell.org/help