Marge Bot pushed to branch master at Glasgow Haskell Compiler / GHC Commits: f6961b02 by Cheng Shao at 2025-11-01T00:08:01+01:00 wasm: reformat dyld source code This commit reformats dyld source code with prettier, to avoid introducing unnecessary diffs in subsequent patches when they're formatted before committing. - - - - - 0c9032a0 by Cheng Shao at 2025-11-01T00:08:01+01:00 wasm: simplify _initialize logic in dyld This commit simplifies how we _initialize a wasm shared library in dyld and removes special treatment for libc.so, see added comment for detailed explanation. - - - - - ec1b40bd by Cheng Shao at 2025-11-01T00:08:01+01:00 wasm: support running dyld fully client side in the browser This commit refactors the wasm dyld script so that it can be used to load and run wasm shared libraries fully client-side in the browser without needing a wasm32-wasi-ghci backend: - A new `DyLDBrowserHost` class is exported, which runs in the browser and uses the in-memory vfs without any RPC calls. This meant to be used to create a `rpc` object for the fully client side use cases. - The exported `main` function now can be used to load user-specified shared libraries, and the user can use the returned `DyLD` instance to run their own exported Haskell functions. - The in-browser wasi implementation is switched to https://github.com/haskell-wasm/browser_wasi_shim for bugfixes and major performance improvements not landed upstream yet. - When being run by deno, it now correctly switches to non-nodejs code paths, so it's more convenient to test dyld logic with deno. See added comments for details, as well as the added `playground001` test case for an example of using it to build an in-browser Haskell playground. - - - - - 8f3e481f by Cheng Shao at 2025-11-01T00:08:01+01:00 testsuite: add playground001 to test haskell playground This commit adds the playground001 test case to test the haskell playground in browser, see comments for details. - - - - - af40606a by Cheng Shao at 2025-11-01T00:08:04+01:00 Revert "testsuite: add T26431 test case" This reverts commit 695036686f8c6d78611edf3ed627608d94def6b7. T26431 is now retired, wasm ghc internal-interpreter logic is tested by playground001. - - - - - 10 changed files: - + testsuite/tests/ghc-api-browser/README.md - + testsuite/tests/ghc-api-browser/all.T - + testsuite/tests/ghc-api-browser/index.html - + testsuite/tests/ghc-api-browser/playground001.hs - + testsuite/tests/ghc-api-browser/playground001.js - + testsuite/tests/ghc-api-browser/playground001.sh - testsuite/tests/ghci-wasm/T26431.stdout → testsuite/tests/ghc-api-browser/playground001.stdout - − testsuite/tests/ghci-wasm/T26431.hs - testsuite/tests/ghci-wasm/all.T - utils/jsffi/dyld.mjs Changes: ===================================== testsuite/tests/ghc-api-browser/README.md ===================================== @@ -0,0 +1,124 @@ +# The Haskell playground browser test + +This directory contains the `playground001` test, which builds a fully +client side Haskell playground in the browser, then runs a +puppeteer-based test to actually interpret a Haskell program in a +headless browser. + +## Headless testing + +`playground001` is tested in GHC CI. To test it locally, first ensure +you've set up the latest +[`ghc-wasm-meta`](https://gitlab.haskell.org/haskell-wasm/ghc-wasm-meta) +toolchain and sourced the `~/.ghc-wasm/env` script, so the right +`node` with the right pre-installed libraries are used. Additionally, +you need to install latest Firefox and: + +```sh +export FIREFOX_LAUNCH_OPTS='{"browser":"firefox","executablePath":"/usr/bin/firefox"}'` +``` + +Or on macOS: + +```sh +export FIREFOX_LAUNCH_OPTS='{"browser":"firefox","executablePath":"/Applications/Firefox.app/Contents/MacOS/firefox"}' +``` + +Without `FIREFOX_LAUNCH_OPTS`, `playground001` is skipped. + +It's possible to test against Chrome as well, the +[`playground001.js`](./playground001.js) test driver doesn't assume +anything Firefox-specific, it just takes the +[`puppeteer.launch`](https://pptr.dev/api/puppeteer.puppeteernode.launch) +options as JSON passed via command line. + +`playground001` works on latest versions of Firefox/Chrome/Safari. + +## Manual testing + +The simplest way to build the playground manually and run it in a +browser tab is to test it once with `--only=playground001 +--keep-test-files` passed to Hadrian, then you can find the temporary +directory containing [`index.html`](./index.html), `rootfs.tar.zst` +etc, then fire up a dev web server and load it. + +Additionally, you can build the playground in tree without invoking +the GHC testsuite. Just build GHC with the wasm target first, then +copy `utils/jsffi/*.mjs` here and run +[`./playground001.sh`](./playground001.sh) script. You need to set +`TEST_CC` to the path of `wasm32-wasi-clang` and `TEST_HC` to the path +of `wasm32-wasi-ghc`, that's it. + +## Customized Haskell playground + +You may want to build a customized Haskell playground that uses GHC +API to interpret Haskell code with custom packages, here are some tips +to get started: + +- Read the code in this directory and figure out how `playground001` + itself works. +- [`./playground001.sh`](./playground001.sh) can be used as a basis to + write your own build/test script. + +You don't need to read the full `dyld.mjs` script. The user-facing +things that are relevant to the playground use case are: + +- `export class DyLDBrowserHost`: it is the `rpc` object required when + calling `main`. You need to pass `stdout`/`stderr` callbacks to + write each line of stdout/stderr, as well as a `rootfs` object that + represents an in-memory vfs containing the shared libraries to load. +- `export async function main`: it eventually returns a `DyLD` object + that can be used like `await + dyld.exportFuncs.myExportedHaskellFunc(js_foo, js_bar)` to invoke + your exported Haskell function. + +Check the source code of [`index.html`](./index.html) and cross +reference [`playground001.hs`](./playground001.hs) for the example of +how they are used. + +The `rootfs` object is a +[`PreopenDirectory`](https://github.com/haskell-wasm/browser_wasi_shim/blob/master/src/fs_mem.ts) +object in the +[`browser_wasi_shim`](https://github.com/haskell-wasm/browser_wasi_shim) +library. The Haskell playground needs a complex vfs containing many +files (shared libraries, interface files, package databases, etc), so +to speed things up, the whole vfs is compressed into a +`rootfs.tar.zst` archive, then that archive is extracted using +[`bsdtar-wasm`](https://github.com/haskell-wasm/bsdtar-wasm). + +You don't need to read the source code of `browser_wasi_shim`; you can +simply paste and adapt the relevant code snippet in +[`index.html`](./index.html) to create the right `rootfs` object from +a tarball. + +The main concern is what do you need to pack into `rootfs.tar.zst`. +For `playground001`, it contains: + +- `/tmp/clib`: the C/C++ shared libraries +- `/tmp/hslib/lib`: the GHC libdir +- `/tmp/libplayground001.so`: the main shared library to start loading + that exports `myMain` + +You can read [`./playground001.sh`](./playground001.sh) to figure out +the details of how I prepare `rootfs.tar.zst` and trim unneeded files +to minimize the tarball size. + +There are multiple possible ways to install third-party packages in +the playground: + +- Start from a `wasm32-wasi-ghc` installation, use `wasm32-wasi-cabal + v1-install --global` to install everything to the global package + database. In theory this is the simplest way, though I haven't tried + it myself and it's unclear to what extent do `v1` commands work + these days. +- Use default nix-style installation, then package the cabal store and + `dist-newstyle` directories into `rootfs.tar.zst`, and pass the + right package database flags when calling GHC API. + +Note that cabal built packages are not relocatable! So things will +break if you build them at a host location and then package into a +different absolute path into the rootfs, keep this in mind. + +If you have any difficulties, you're welcome to the [Haskell +Wasm](https://matrix.to/#/#haskell.wasm:matrix.org) matrix room for +community support. ===================================== testsuite/tests/ghc-api-browser/all.T ===================================== @@ -0,0 +1,52 @@ +# makefile_test/run_command is skipped when config.target_wrapper is +# not None, see test_common_work in testsuite/driver/testlib.py. for +# now just use this workaround to run custom test script here; ideally +# we'd fix test failures elsewhere and enable +# makefile_test/run_command for cross targets some day. +async def stub_run_command(name, way, cmd): + return await run_command(name, way, cmd) + + +# config.target_wrapper is prepended when running any command when +# testing a cross target, see simple_run in +# testsuite/driver/testlib.py. this is problematic when running a host +# test script. for now do this override; ideally we'd have clear +# host/target distinction for command invocations in the testsuite +# driver instead of just a command string. +def override_target_wrapper(name, opts): + opts.target_wrapper = "" + + +setTestOpts( + [ + unless(arch("wasm32"), skip), + override_target_wrapper, + high_memory_usage, + ignore_stderr, + only_ways(["dyn"]), + extra_ways(["dyn"]), + ] +) + + +test( + "playground001", + [ + # pretty heavyweight, just test one browser for now. + unless("FIREFOX_LAUNCH_OPTS" in ghc_env, skip), + extra_files( + [ + "../../../.gitlab/hello.hs", + "../../../utils/jsffi/dyld.mjs", + "../../../utils/jsffi/post-link.mjs", + "../../../utils/jsffi/prelude.mjs", + "index.html", + "playground001.hs", + "playground001.js", + "playground001.sh", + ] + ), + ], + stub_run_command, + ['./playground001.sh "$FIREFOX_LAUNCH_OPTS"'], +) ===================================== testsuite/tests/ghc-api-browser/index.html ===================================== @@ -0,0 +1,234 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1" /> + <title>ghc-in-browser</title> + <link + rel="stylesheet" + href="https://cdn.jsdelivr.net/npm/modern-normalize/modern-normalize.min.css" + /> + <style> + html, + body { + height: 100%; + } + body { + margin: 0; + font-family: system-ui, -apple-system, Segoe UI, Roboto, sans-serif; + background: #0f172a; + color: #e5e7eb; + } + .app { + height: 100vh; + display: grid; + gap: 0.5rem; + padding: 0.5rem; + } + @media (min-width: 800px) { + .app { + grid-template-columns: 1fr 1fr; + } + } + @media (max-width: 799.98px) { + .app { + grid-template-rows: 1fr 1fr; + } + } + .pane { + background: #111827; + border: 1px solid #1f2937; + border-radius: 12px; + display: flex; + flex-direction: column; + min-height: 0; + } + header { + padding: 0.5rem 0.75rem; + border-bottom: 1px solid #1f2937; + font-weight: 600; + } + #editor { + flex: 1; + min-height: 0; + } + .right { + padding: 0.6rem; + gap: 0.6rem; + } + .controls { + display: flex; + gap: 0.5rem; + flex-wrap: wrap; + margin-bottom: 0.4rem; + } + .controls input[type="text"] { + flex: 1; + min-width: 200px; + background: #0b1020; + color: #e5e7eb; + border: 1px solid #223; + border-radius: 8px; + padding: 0.55rem; + } + .controls button { + background: #22c55e; + border: none; + border-radius: 8px; + padding: 0.55rem 0.85rem; + font-weight: 600; + cursor: pointer; + } + .outputs { + display: block; + } + .outputs .label { + font-size: 0.85rem; + opacity: 0.8; + margin: 0.35rem 0; + } + .outputs textarea { + display: block; + width: 100%; + min-height: 30vh; + background: #0b1020; + color: #d1fae5; + border: 1px solid #223; + border-radius: 8px; + padding: 0.6rem; + resize: vertical; + } + .stderr { + color: #fee2e2; + } + </style> + + <script async type="module"> + import * as monaco from "https://cdn.jsdelivr.net/npm/monaco-editor/+esm"; + import { + ConsoleStdout, + File, + OpenFile, + PreopenDirectory, + WASI, + } from "https://esm.sh/gh/haskell-wasm/browser_wasi_shim"; + import { DyLDBrowserHost, main } from "./dyld.mjs"; + + const rootfs = new PreopenDirectory("/", []); + + const bsdtar_wasi = new WASI( + ["bsdtar.wasm", "-x"], + [], + [ + new OpenFile(new File(new Uint8Array(), { readonly: true })), + ConsoleStdout.lineBuffered((msg) => console.info(msg)), + ConsoleStdout.lineBuffered((msg) => console.warn(msg)), + rootfs, + ], + { debug: false } + ); + + const [{ instance }, rootfs_bytes] = await Promise.all([ + WebAssembly.instantiateStreaming( + fetch("https://haskell-wasm.github.io/bsdtar-wasm/bsdtar.wasm"), + { wasi_snapshot_preview1: bsdtar_wasi.wasiImport } + ), + fetch("./rootfs.tar.zst").then((r) => r.bytes()), + ]); + + bsdtar_wasi.fds[0] = new OpenFile( + new File(rootfs_bytes, { readonly: true }) + ); + bsdtar_wasi.start(instance); + + if (document.readyState === "loading") { + await new Promise((res) => + document.addEventListener("DOMContentLoaded", res, { once: true }) + ); + } + + window.editor = monaco.editor.create(document.getElementById("editor"), { + value: 'main :: IO ()\nmain = putStrLn "Hello, Haskell!"\n', + language: "haskell", + automaticLayout: true, + minimap: { enabled: false }, + theme: "vs-dark", + fontSize: 14, + }); + + const dyld = await main({ + rpc: new DyLDBrowserHost({ + rootfs, + stdout: (msg) => { + document.getElementById("stdout").value += `${msg}\n`; + }, + stderr: (msg) => { + document.getElementById("stderr").value += `${msg}\n`; + }, + }), + searchDirs: [ + "/tmp/clib", + "/tmp/hslib/lib/wasm32-wasi-ghc-9.15.20251024", + ], + mainSoPath: "/tmp/libplayground001.so", + args: ["libplayground001.so", "+RTS", "-c", "-RTS"], + isIserv: false, + }); + const main_func = await dyld.exportFuncs.myMain("/tmp/hslib/lib"); + + document.getElementById("runBtn").addEventListener("click", async () => { + document.getElementById("runBtn").disabled = true; + + try { + document.getElementById("stdout").value = ""; + document.getElementById("stderr").value = ""; + + await main_func( + document.getElementById("ghcArgs").value, + editor.getValue() + ); + } finally { + document.getElementById("runBtn").disabled = false; + } + }); + + document.getElementById("runBtn").disabled = false; + </script> + </head> + <body> + <div class="app"> + <section class="pane"> + <header>Haskell Source</header> + <div id="editor"></div> + </section> + + <section class="pane right"> + <header>Controls / Output</header> + <div class="controls"> + <input + id="ghcArgs" + type="text" + placeholder="GHC args" + style="font-family: ui-monospace, Menlo, Consolas, monospace" + /> + <button id="runBtn" disabled="true">Run</button> + </div> + <div class="outputs"> + <div class="label">stdout</div> + <textarea + id="stdout" + readonly + style="font-family: ui-monospace, Menlo, Consolas, monospace" + ></textarea> + <div class="label">stderr</div> + <textarea + id="stderr" + class="stderr" + readonly + style="font-family: ui-monospace, Menlo, Consolas, monospace" + ></textarea> + </div> + </section> + </div> + </body> +</html> ===================================== testsuite/tests/ghc-api-browser/playground001.hs ===================================== @@ -0,0 +1,95 @@ +module Playground + ( myMain, + ) +where + +import Control.Monad +import Data.Coerce +import Data.IORef +import GHC +import GHC.Driver.Config.Diagnostic +import GHC.Driver.Errors +import GHC.Driver.Errors.Types +import GHC.Driver.Monad +import GHC.Plugins +import GHC.Runtime.Interpreter +import GHC.Utils.Exception +import GHC.Wasm.Prim + +newtype JSFunction t = JSFunction JSVal + +type ExportedMainFunction = JSString -> JSString -> IO () + +-- main entry point of playground001, returns a js async function that +-- takes ghc args and Main.hs content, interprets Main.hs and runs +-- Main.main. +myMain :: JSString -> IO (JSFunction ExportedMainFunction) +myMain js_libdir = + defaultErrorHandler defaultFatalMessager defaultFlushOut $ do + libdir <- evaluate $ fromJSString js_libdir + freeJSVal $ coerce js_libdir + -- we don't use runGhc since we want to share a session to be + -- reused. + session <- Session <$> newIORef undefined + -- save a fresh default dflags, otherwise user input ghc args are + -- not properly reset. + dflags0 <- flip reflectGhc session $ do + initGhcMonad (Just libdir) + dflags0 <- getSessionDynFlags + setSessionDynFlags $ + dflags0 + { ghcMode = CompManager, + backend = bytecodeBackend, + ghcLink = LinkInMemory, + verbosity = 1 + } + getSessionDynFlags + -- this is always run in a forked thread. which is fine as long as + -- the sesssion is not reused concurrently, but it's up to the + -- caller in js to ensure that. we simply disable the run button + -- until each run completes in the playground ui logic. + toMainFunc $ \js_args js_src -> + defaultErrorHandler defaultFatalMessager defaultFlushOut $ do + args <- evaluate $ words $ fromJSString js_args + freeJSVal $ coerce js_args + writeFile f $ fromJSString js_src + freeJSVal $ coerce js_src + -- it's fine to call withCleanupSession since it just cleans up + -- tmpfs for now. in the future if it does more cleanup that + -- makes the session state invalid for reuse, just remove it; + -- everything will be cleaned up anyway when the browser tab is + -- closed + flip reflectGhc session $ withCleanupSession $ do + setSessionDynFlags dflags0 + logger0 <- getLogger + (dflags1, _, dynamicFlagWarnings) <- + parseDynamicFlags logger0 dflags0 $ map noLoc args + setSessionDynFlags dflags1 + logger1 <- getLogger + liftIO + $ printOrThrowDiagnostics + logger1 + (initPrintConfig dflags1) + (initDiagOpts dflags1) + $ GhcDriverMessage + <$> dynamicFlagWarnings + setTargets =<< (: []) <$> guessTarget f Nothing Nothing + r <- load LoadAllTargets + when (failed r) $ fail "load returned Failed" + setContext [IIDecl $ simpleImportDecl $ mkModuleName "Main"] + fhv <- compileExprRemote "Main.main" + hsc_env <- getSession + liftIO $ evalIO (hscInterp hsc_env) fhv + where + f = "/tmp/Main.hs" + +foreign import javascript "wrapper" + toMainFunc :: + ExportedMainFunction -> + IO (JSFunction ExportedMainFunction) + +foreign export javascript "myMain" + myMain :: + JSString -> + IO + (JSFunction ExportedMainFunction) ===================================== testsuite/tests/ghc-api-browser/playground001.js ===================================== @@ -0,0 +1,91 @@ +#!/usr/bin/env -S node + +const puppeteer = require("puppeteer-core"); +const fs = require("node:fs"); +const path = require("node:path"); + +class Playground { + static #token = Symbol("Playground"); + #browser; + #page; + + static async create({ launchOpts, artifactDir }) { + const playground = new Playground(Playground.#token); + playground.#browser = await puppeteer.launch(launchOpts); + + playground.#page = await playground.#browser.newPage(); + await playground.#page.setRequestInterception(true); + playground.#page.on("request", async (req) => { + if (!req.url().startsWith("http://localhost")) { + return req.continue(); + } + + try { + const f = req.url().replace("http://localhost", artifactDir); + return req.respond({ + status: 200, + contentType: + { + ".html": "text/html", + ".mjs": "application/javascript", + }[path.extname(f)] || "application/octet-stream", + body: await fs.promises.readFile(f), + }); + } catch { + return req.abort(); + } + }); + + await playground.#page.goto("http://localhost/index.html"); + await playground.#page.locator("#runBtn:enabled").wait(); + return playground; + } + + async close() { + await this.#browser.close(); + } + + async runMain({ mainSrc, ghcArgs }) { + await Promise.all([ + this.#page.evaluate((mainSrc) => editor.setValue(mainSrc), mainSrc), + this.#page.locator("#ghcArgs").fill(ghcArgs), + ]); + await this.#page.locator("#runBtn:enabled").click(); + await this.#page.locator("#runBtn:enabled").wait(); + + const [stdout, stderr] = await Promise.all( + ["#stdout", "#stderr"].map((el) => + this.#page + .locator(el) + .map((t) => t.value) + .wait() + ) + ); + + return { stdout, stderr }; + } + + constructor(token) { + if (token !== Playground.#token) { + throw new Error("new Playground() is forbidden, use Playground.create()"); + } + } +} + +(async () => { + const playground = await Playground.create({ + launchOpts: JSON.parse(process.argv[2]), + artifactDir: process.cwd(), + }); + + try { + const { stdout, stderr } = await playground.runMain({ + mainSrc: await fs.promises.readFile("./hello.hs", { encoding: "utf-8" }), + ghcArgs: "-package ghc -v0", + }); + process.stdout.write(stdout); + process.stderr.write(stderr); + } finally { + await playground.close(); + } +})(); ===================================== testsuite/tests/ghc-api-browser/playground001.sh ===================================== @@ -0,0 +1,76 @@ +#!/usr/bin/env bash + +set -euo pipefail + +# also set this when building wasm32-wasi-ghc for production +# deployment of haskell playground, so all the .so files are +# optimized. +export WASM_SO_OPT="--debuginfo --low-memory-unused --strip-dwarf -Oz" + +# we'll build a rootfs tarball that contains everything in tmp and +# extracts to /tmp, mapped from here +mkdir ./tmp + +$TEST_HC \ + -v0 \ + -package ghc \ + -shared -dynamic \ + -no-keep-hi-files -no-keep-o-files \ + -O2 \ + playground001.hs -o ./tmp/libplayground001.so +rm -f ./*_stub.h ./playground001.hs + +# /tmp/clib contains libc/libc++ .so files +cp -r "$(dirname "$TEST_CC")/../share/wasi-sysroot/lib/wasm32-wasi" ./tmp/clib +# trim unneeded stuff in c libdir +find ./tmp/clib -type f ! -name "*.so" -delete +rm -f \ + ./tmp/clib/libsetjmp.so \ + ./tmp/clib/libwasi-emulated-*.so + +# /tmp/hslib/lib is the ghc libdir +mkdir ./tmp/hslib +cp -r "$($TEST_HC --print-libdir)" ./tmp/hslib/lib +# unregister Cabal/Cabal-syntax, too big +$GHC_PKG --no-user-package-db --global-package-db=./tmp/hslib/lib/package.conf.d unregister Cabal Cabal-syntax +$GHC_PKG --no-user-package-db --global-package-db=./tmp/hslib/lib/package.conf.d recache +# we only need non-profiling .dyn_hi/.so, trim as much as we can +find ./tmp/hslib/lib "(" \ + -name "*.hi" \ + -o -name "*.a" \ + -o -name "*.p_hi" \ + -o -name "libHS*_p.a" \ + -o -name "*.p_dyn_hi" \ + -o -name "libHS*_p*.so" \ + -o -name "libHSrts*_debug*.so" \ + ")" -delete +rm -rf \ + ./tmp/hslib/lib/doc \ + ./tmp/hslib/lib/html \ + ./tmp/hslib/lib/latex \ + ./tmp/hslib/lib/*.mjs \ + ./tmp/hslib/lib/*.js \ + ./tmp/hslib/lib/*.txt +# HS_SEARCHDIR is something like +# /tmp/hslib/lib/wasm32-wasi-ghc-9.15.20251024 which is the +# dynamic-library-dirs that contains all libHS*.so in one place, and +# also static libraries in per-unit directories +HS_SEARCHDIR=$(find ./tmp/hslib/lib -type f -name "*.so" -print0 | xargs -0 -n1 dirname | sort -u | sed "s|^\./|/|") +# hunt down the remaining bits of Cabal/Cabal-syntax. too bad there's +# no ghc-pkg uninstall. +rm -rf ."$HS_SEARCHDIR"/*Cabal* + +# fix the hard coded search dir in index.html +SED_IS_GNU=$(sed --version &> /dev/null && echo 1 || echo 0) +if [[ $SED_IS_GNU == "1" ]]; then + sed -i "s|/tmp/hslib/lib/wasm32-wasi-ghc-9.15.20251024|$HS_SEARCHDIR|" ./index.html +else + sed -i "" "s|/tmp/hslib/lib/wasm32-wasi-ghc-9.15.20251024|$HS_SEARCHDIR|" ./index.html +fi + +# also set ZSTD_NBTHREADS/ZSTD_CLEVEL when building for production +tar -cf ./rootfs.tar.zst --zstd tmp +rm -rf ./tmp + +# pass puppeteer.launch() opts as json +exec ./playground001.js "$1" ===================================== testsuite/tests/ghci-wasm/T26431.stdout → testsuite/tests/ghc-api-browser/playground001.stdout ===================================== ===================================== testsuite/tests/ghci-wasm/T26431.hs deleted ===================================== @@ -1,35 +0,0 @@ -import Control.Exception -import Control.Monad.IO.Class -import Data.Maybe -import GHC -import GHC.Plugins -import GHC.Runtime.Interpreter -import System.Environment.Blank - -main :: IO () -main = do - [libdir] <- getArgs - defaultErrorHandler defaultFatalMessager defaultFlushOut $ - runGhc (Just libdir) $ - do - dflags0 <- getSessionDynFlags - let dflags1 = - dflags0 - { ghcMode = CompManager, - backend = bytecodeBackend, - ghcLink = LinkInMemory - } - logger <- getLogger - (dflags2, _, _) <- - parseDynamicFlags logger dflags1 $ - map noLoc ["-package", "ghc"] - _ <- setSessionDynFlags dflags2 - addTarget =<< guessTarget "hello.hs" Nothing Nothing - _ <- load LoadAllTargets - setContext - [ IIDecl $ simpleImportDecl $ mkModuleName "Prelude", - IIDecl $ simpleImportDecl $ mkModuleName "Main" - ] - hsc_env <- getSession - fhv <- compileExprRemote "main" - liftIO $ evalIO (fromJust $ hsc_interp hsc_env) fhv ===================================== testsuite/tests/ghci-wasm/all.T ===================================== @@ -10,11 +10,3 @@ test('T26430', [ extra_hc_opts('-L. -lT26430B')] , compile_and_run, [''] ) - -test('T26431', [ - extra_files(['../../../.gitlab/hello.hs']), - extra_hc_opts('-package ghc'), - extra_run_opts(f'"{config.libdir}"'), - ignore_stderr] -, compile_and_run, [''] -) ===================================== utils/jsffi/dyld.mjs ===================================== @@ -285,7 +285,7 @@ function originFromServerAddress({ address, family, port }) { } // Browser/node portable code stays above this watermark. -const isNode = Boolean(globalThis?.process?.versions?.node); +const isNode = Boolean(globalThis?.process?.versions?.node && !globalThis.Deno); // Too cumbersome to only import at use sites. Too troublesome to // factor out browser-only/node-only logic into different modules. For @@ -307,27 +307,27 @@ if (isNode) { ws = require("ws"); } catch {} } else { - wasi = await import( - "https://cdn.jsdelivr.net/npm/@bjorn3/browser_wasi_shim@0.4.2/dist/index.js" - ); + wasi = await import("https://esm.sh/gh/haskell-wasm/browser_wasi_shim"); } // A subset of dyld logic that can only be run in the host node // process and has full access to local filesystem -class DyLDHost { +export class DyLDHost { // Deduped absolute paths of directories where we lookup .so files #rpaths = new Set(); - constructor() { - // Inherited pipe file descriptors from GHC - const out_fd = Number.parseInt(process.argv[4]), - in_fd = Number.parseInt(process.argv[5]); - + constructor({ outFd, inFd }) { + // When running a non-iserv shared library with node, the DyLDHost + // instance is created without a pair of fds, so skip creation of + // readStream/writeStream, they won't be used anyway + if (!(typeof outFd === "number" && typeof inFd === "number")) { + return; + } this.readStream = stream.Readable.toWeb( - fs.createReadStream(undefined, { fd: in_fd }) + fs.createReadStream(undefined, { fd: inFd }) ); this.writeStream = stream.Writable.toWeb( - fs.createWriteStream(undefined, { fd: out_fd }) + fs.createWriteStream(undefined, { fd: outFd }) ); } @@ -377,6 +377,72 @@ class DyLDHost { } } +// Runs in the browser and uses the in-memory vfs, doesn't do any RPC +// calls +export class DyLDBrowserHost { + // Deduped absolute paths of directories where we lookup .so files + #rpaths = new Set(); + // The PreopenDirectory object of the root filesystem + rootfs; + // Continuations to output a single line to stdout/stderr + stdout; + stderr; + + // Given canonicalized absolute file path, returns the File object, + // or null if absent + #readFile(p) { + const { ret, entry } = this.rootfs.dir.get_entry_for_path({ + parts: p.split("/").filter((tok) => tok !== ""), + is_dir: false, + }); + return ret === 0 ? entry : null; + } + + constructor({ rootfs, stdout, stderr }) { + this.rootfs = rootfs + ? rootfs + : new wasi.PreopenDirectory("/", [["tmp", new wasi.Directory([])]]); + this.stdout = stdout ? stdout : (msg) => console.info(msg); + this.stderr = stderr ? stderr : (msg) => console.warn(msg); + } + + // p must be canonicalized absolute path + async addLibrarySearchPath(p) { + this.#rpaths.add(p); + return null; + } + + async findSystemLibrary(f) { + if (f.startsWith("/")) { + if (this.#readFile(f)) { + return f; + } + throw new Error(`findSystemLibrary(${f}): not found in /`); + } + + for (const rpath of this.#rpaths) { + const r = `${rpath}/${f}`; + if (this.#readFile(r)) { + return r; + } + } + + throw new Error( + `findSystemLibrary(${f}): not found in ${[...this.#rpaths]}` + ); + } + + async fetchWasm(p) { + const entry = this.#readFile(p); + const r = new Response(entry.data, { + headers: { "Content-Type": "application/wasm" }, + }); + // It's only fetched once, take the chance to prune it in vfs to save memory + entry.data = new Uint8Array(); + return r; + } +} + // Fulfill the same functionality as DyLDHost by doing fetch() calls // to respective RPC endpoints of a host http server. Also manages // WebSocket connections back to host. @@ -494,7 +560,7 @@ export class DyLDRPC { // Actual implementation of endpoints used by DyLDRPC class DyLDRPCServer { - #dyldHost = new DyLDHost(); + #dyldHost; #server; #wss; @@ -502,11 +568,15 @@ class DyLDRPCServer { host, port, dyldPath, - libdir, - ghciSoPath, + searchDirs, + mainSoPath, + outFd, + inFd, args, redirectWasiConsole, }) { + this.#dyldHost = new DyLDHost({ outFd, inFd }); + this.#server = http.createServer(async (req, res) => { const origin = originFromServerAddress(await this.listening); @@ -540,7 +610,7 @@ class DyLDRPCServer { res.end( ` import { DyLDRPC, main } from "./fs${dyldPath}"; -const args = ${JSON.stringify({ libdir, ghciSoPath, args })}; +const args = ${JSON.stringify({ searchDirs, mainSoPath, args, isIserv: true })}; args.rpc = new DyLDRPC({origin: "${origin}", redirectWasiConsole: ${redirectWasiConsole}}); args.rpc.opened.then(() => main(args)); ` @@ -829,11 +899,37 @@ class DyLD { ), wasi.ConsoleStdout.lineBuffered((msg) => this.#rpc.stdout(msg)), wasi.ConsoleStdout.lineBuffered((msg) => this.#rpc.stderr(msg)), + // for ghci browser mode, default to an empty rootfs with + // /tmp + this.#rpc instanceof DyLDBrowserHost + ? this.#rpc.rootfs + : new wasi.PreopenDirectory("/", [["tmp", new wasi.Directory([])]]), ], { debug: false } ); } + // Both wasi implementations we use provide + // wasi.initialize(instance) to initialize a wasip1 reactor + // module. However, instance does not really need to be a + // WebAssembly.Instance object; the wasi implementations only need + // to access instance.exports.memory for the wasi syscalls to + // work. + // + // Given we'll reuse the same wasi object across different + // WebAssembly.Instance objects anyway and + // wasi.initialize(instance) can't be called more than once, we + // use this simple trick and pass a fake instance object that + // contains just enough info for the wasi implementation to + // initialize its internal state. Later when we load each wasm + // shared library, we can just manually invoke their + // initialization functions. + this.#wasi.initialize({ + exports: { + memory: this.#memory, + }, + }); + // Keep this in sync with rts/wasm/Wasm.S! for (let i = 1; i <= 10; ++i) { this.#regs[`__R${i}`] = new WebAssembly.Global({ @@ -930,10 +1026,15 @@ class DyLD { async loadDLLs(packed) { // Normalize input to an array of strings. When called from Haskell // we pass a single JSString containing NUL-separated paths. - const paths = (typeof packed === "string" - ? (packed.length === 0 ? [] : packed.split("\0")) - : [packed] // tolerate an accidental single path object - ).filter((s) => s.length > 0).reverse(); + const paths = ( + typeof packed === "string" + ? packed.length === 0 + ? [] + : packed.split("\0") + : [packed] + ) // tolerate an accidental single path object + .filter((s) => s.length > 0) + .reverse(); // Compute a single downsweep plan for the whole batch. // Note: #downsweep mutates #loadedSos to break cycles and dedup. @@ -1154,22 +1255,6 @@ class DyLD { throw new Error(`cannot handle export ${k} ${v}`); } - // We call wasi.initialize when loading libc.so, then reuse the - // wasi instance globally. When loading later .so files, just - // manually invoke _initialize(). - if (soname === "libc.so") { - instance.exports.__wasm_apply_data_relocs(); - // wasm-ld forbits --export-memory with --shared, I don't know - // why but this is sufficient to make things work - this.#wasi.initialize({ - exports: { - memory: this.#memory, - _initialize: instance.exports._initialize, - }, - }); - continue; - } - // See // https://gitlab.haskell.org/haskell-wasm/llvm-project/-/blob/release/21.x/lld..., // __wasm_apply_data_relocs is now optional so only call it if @@ -1180,7 +1265,7 @@ class DyLD { // been called upon instantiation, see // Writer::createStartFunction(). if (instance.exports.__wasm_apply_data_relocs) { - instance.exports.__wasm_apply_data_relocs(); + instance.exports.__wasm_apply_data_relocs(); } instance.exports._initialize(); @@ -1208,15 +1293,38 @@ class DyLD { } } -export async function main({ rpc, libdir, ghciSoPath, args }) { +// The main entry point of dyld that may be run on node/browser, and +// may run either iserv defaultMain from the ghci library or an +// alternative entry point from another shared library +export async function main({ + rpc, // Handle the side effects of DyLD + searchDirs, // Initial library search directories + mainSoPath, // Could also be another shared library that's actually not ghci + args, // WASI argv starting with the executable name. +RTS etc will be respected + isIserv, // set to true when running iserv defaultServer +}) { try { const dyld = new DyLD({ - args: ["dyld.so", ...args], + args, rpc, }); - await dyld.addLibrarySearchPath(libdir); - await dyld.loadDLLs(ghciSoPath); + for (const libdir of searchDirs) { + await dyld.addLibrarySearchPath(libdir); + } + await dyld.loadDLLs(mainSoPath); + + // At this point, rts/ghc-internal are loaded, perform wasm shared + // library specific RTS startup logic, see Note [JSFFI initialization] + dyld.exportFuncs.__ghc_wasm_jsffi_init(); + + // We're not running iserv, just return the dyld instance so user + // could use it to invoke their exported functions, and don't + // perform cleanup (see finally block) + if (!isIserv) { + return dyld; + } + // iserv-specific logic follows const reader = rpc.readStream.getReader(); const writer = rpc.writeStream.getWriter(); @@ -1235,31 +1343,25 @@ export async function main({ rpc, libdir, ghciSoPath, args }) { writer.write(new Uint8Array(buf)); }; - dyld.exportFuncs.__ghc_wasm_jsffi_init(); - await dyld.exportFuncs.defaultServer(cb_sig, cb_recv, cb_send); + return await dyld.exportFuncs.defaultServer(cb_sig, cb_recv, cb_send); } finally { - rpc.close(); + if (isIserv) { + rpc.close(); + } } } -(async () => { - if (!isNode) { - return; - } - - const libdir = process.argv[2]; - const ghciSoPath = process.argv[3]; - const args = process.argv.slice(6); - +// node-specific iserv-specific logic +async function nodeMain({ searchDirs, mainSoPath, outFd, inFd, args }) { if (!process.env.GHCI_BROWSER) { - const rpc = new DyLDHost(); - await main({ + const rpc = new DyLDHost({ outFd, inFd }); + return await main({ rpc, - libdir, - ghciSoPath, + searchDirs, + mainSoPath, args, + isIserv: true, }); - return; } if (!ws) { @@ -1272,8 +1374,10 @@ export async function main({ rpc, libdir, ghciSoPath, args }) { host: process.env.GHCI_BROWSER_HOST || "127.0.0.1", port: process.env.GHCI_BROWSER_PORT || 0, dyldPath: import.meta.filename, - libdir, - ghciSoPath, + searchDirs, + mainSoPath, + outFd, + inFd, args, redirectWasiConsole: process.env.GHCI_BROWSER_PUPPETEER_LAUNCH_OPTS || @@ -1362,6 +1466,20 @@ export async function main({ rpc, libdir, ghciSoPath, args }) { } console.log( - `Open ${origin}/main.html or import ${origin}/main.js to boot ghci` + `Open ${origin}/main.html or import("${origin}/main.js") to boot ghci` ); -})(); +} + +const isNodeMain = isNode && import.meta.filename === process.argv[1]; + +// node iserv as invoked by +// GHC.Runtime.Interpreter.Wasm.spawnWasmInterp +if (isNodeMain) { + const clibdir = process.argv[2]; + const mainSoPath = process.argv[3]; + const outFd = Number.parseInt(process.argv[4]), + inFd = Number.parseInt(process.argv[5]); + const args = ["dyld.so", ...process.argv.slice(6)]; + + await nodeMain({ searchDirs: [clibdir], mainSoPath, outFd, inFd, args }); +} View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/995dfe0d6012c2798bafe42cabbc04b... -- View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/995dfe0d6012c2798bafe42cabbc04b... You're receiving this email because of your account on gitlab.haskell.org.