[Git][ghc/ghc][master] JS: remove archive padding after wasm objects (#25665)
Marge Bot pushed to branch master at Glasgow Haskell Compiler / GHC Commits: 593f5f5b by Sylvain Henry at 2026-10-22T04:51:16-04:00 JS: remove archive padding after wasm objects (#25665) On Darwin hosts, emar (llvm-ar) writes archives in the Darwin format when the first member is not a known object file, which is the case for the archives of the JS backend. This format pads each member to 8 bytes with '\n' and includes the padding in the member size. Wasm objects do not accept trailing bytes: wasm-ld failed with "section too large". The JS linker now removes this padding when it extracts a wasm object from an archive. In addition, Hadrian passes --format=gnu to emar for the JS target. See Note [Wasm objects in Darwin archives]. Also document the format of Cc (wasm) objects in Note [Cc object format]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> - - - - - 12 changed files: - + changelog.d/25665 - compiler/GHC/StgToJS/Linker/Linker.hs - compiler/GHC/StgToJS/Object.hs - hadrian/src/Settings/Builders/Ar.hs - testsuite/tests/javascript/Makefile - + testsuite/tests/javascript/T25665.hs - + testsuite/tests/javascript/T25665.pkg - + testsuite/tests/javascript/T25665.stdout - + testsuite/tests/javascript/T25665Lib.hs - + testsuite/tests/javascript/T25665_c.c - + testsuite/tests/javascript/T25665_js.js - testsuite/tests/javascript/all.T Changes: ===================================== changelog.d/25665 ===================================== @@ -0,0 +1,6 @@ +section: js-backend +synopsis: Fix linking of C sources in packages on Darwin (``wasm-ld: section too large``). + The padding added by ``llvm-ar`` after wasm objects in Darwin format archives is now removed. + Hadrian now also builds the archives of the JS boot libraries in the GNU format. +mrs: !16809 +issues: #25665 ===================================== compiler/GHC/StgToJS/Linker/Linker.hs ===================================== @@ -269,9 +269,10 @@ jsLink lc_cfg cfg logger tmpfs ar_cache out link_plan = do go_entries emcc_opts cc_objs es Just ObjCc -> do -- extract the object file from the archive in a temporary - -- file and return its path + -- file and return its path. + -- See Note [Wasm objects in Darwin archives] in GHC.StgToJS.Object cc_obj_fn <- newTempName logger tmpfs tmp_dir TFL_CurrentModule "o" - B.writeFile cc_obj_fn (Ar.filedata e) + B.writeFile cc_obj_fn (trimWasmObjectBS (Ar.filedata e)) let cc_objs' = cc_obj_fn:cc_objs go_entries emcc_opts cc_objs' es Just ObjJs -> do ===================================== compiler/GHC/StgToJS/Object.hs ===================================== @@ -26,6 +26,8 @@ module GHC.StgToJS.Object ( ObjectKind(..) , getObjectKind , getObjectKindBS + -- * Cc object + , trimWasmObjectBS -- * JS object , JSOptions(..) , defaultJSOptions @@ -109,7 +111,8 @@ import qualified Control.Exception as Exception data ObjectKind = ObjJs -- ^ JavaScript source embedded in a .o | ObjHs -- ^ JS backend object for Haskell code - | ObjCc -- ^ Wasm module object as produced by emcc + | ObjCc -- ^ Wasm module object as produced by emcc. + -- See Note [Cc object format] deriving (Show,Eq,Ord) -- | Get the kind of a file object, if any @@ -141,6 +144,120 @@ hsHeader = unsafePerformIO $ B.unsafePackAddressLen 8 "GHCJS_HS"# wasmHeader :: B.ByteString wasmHeader = unsafePerformIO $ B.unsafePackAddressLen 4 "\0asm"# +-- Note [Cc object format] +-- ~~~~~~~~~~~~~~~~~~~~~~~ +-- Cc objects are wasm object files produced by emcc (clang). A wasm object +-- file is a wasm module binary with additional custom sections ("linking", +-- "reloc.*") for the linker. Layout: +-- +-- magic : 4 bytes "\0asm" +-- version : 4 bytes, u32 little-endian (1) +-- sections : until the end of the file. Each section is: +-- id : 1 byte +-- size : u32 as unsigned LEB128 (1 to 5 bytes): size of the contents +-- contents : `size` bytes +-- +-- There is no section count and no total size: a reader reads sections until +-- the end of the file. Thus a reader reads trailing bytes as a section (see +-- Note [Wasm objects in Darwin archives]). +-- +-- Unsigned LEB128: each byte gives 7 bits of the value, least significant +-- group first. Bit 7 is set in all the bytes except the last one. E.g. 300 +-- (0b10_0101100) is encoded as [0xac, 0x02]. +-- +-- We only use the magic (to detect Cc objects, see getObjectKindBS) and the +-- section headers (to find the end of the last section, see +-- trimWasmObjectBS). We do not read the contents of the sections. +-- +-- References: +-- - WebAssembly spec, binary format of modules and sections: +-- https://webassembly.github.io/spec/core/binary/modules.html +-- - WebAssembly spec, LEB128 integers: +-- https://webassembly.github.io/spec/core/binary/values.html +-- - Wasm object files (tool conventions, not in the spec): +-- https://github.com/WebAssembly/tool-conventions/blob/main/Linking.md +-- - LLVM implementation (used by wasm-ld): readSection in +-- llvm/lib/Object/WasmObjectFile.cpp + +-- | Remove the archive padding at the end of a wasm object, if any. +-- +-- See Note [Wasm objects in Darwin archives] and Note [Cc object format] +trimWasmObjectBS :: B.ByteString -> B.ByteString +trimWasmObjectBS bs = go 8 -- skip magic and version + where + len = B.length bs + + go !off + | off >= len + = bs + | Just sz <- full_section_size off + , off + sz <= len + = go (off + sz) + | is_padding (B.drop off bs) + = B.take off bs + | otherwise + = bs -- not padding: keep the object and let wasm-ld report the error + + is_padding rest = B.length rest < 8 && B.all (== 0x0a) rest + + -- full size of the section at the given offset: id, payload size, payload + full_section_size off = do + (payload_sz, n) <- get_uleb128 (off + 1) 0 0 + pure (1 + n + payload_sz) + + -- read a LEB128 u32 (5 bytes max). Return the value and its size in bytes. + get_uleb128 :: Int -> Int -> Int -> Maybe (Int, Int) + get_uleb128 start i acc + | i >= 5 || start + i >= len + = Nothing + | otherwise + = let b = B.index bs (start + i) + !acc' = acc .|. (fromIntegral (b .&. 0x7f) `shiftL` (7 * i)) + in if testBit b 7 + then get_uleb128 start (i + 1) acc' + else Just (acc', i + 1) + +-- Note [Wasm objects in Darwin archives] +-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +-- llvm-ar (used as emar) supports several archive formats: gnu, bsd, darwin, +-- etc. (see the --format option). On Darwin hosts, it uses the Darwin format +-- by default when the first member is not a known object file, which is the +-- case for the JS and HS objects of the JS backend. +-- +-- The Darwin format is a variant of the BSD format: member names are stored +-- in the member data (#1/N names) and the data of each member is aligned to 8 +-- bytes with '\n' (0x0a) bytes. This alignment is what ld64 expects (see +-- computeMemberData in llvm/lib/Object/ArchiveWriter.cpp). The size in the +-- member header includes the name and the padding. The archive parser +-- (GHC.SysTools.Ar) removes the name but not the padding: it returns the +-- padding as part of the member data. The JS linker removes it with +-- trimWasmObjectBS (see below). +-- +-- Only the Darwin format has this padding: the gnu and bsd formats do not. +-- "llvm-ar --format=darwin" reproduces the issue on any host. +-- +-- JS and HS objects accept trailing bytes. A wasm object does not: it is a +-- sequence of sections until the end of the file. +-- See Note [Cc object format]. wasm-ld reads "\n\n" as a section with id 10 +-- and payload size 10 and fails with "section too large" (#25665). +-- +-- Thus, before we give a wasm object from an archive to emcc, the JS linker +-- calls trimWasmObjectBS: we read the sections of the object and we remove the +-- trailing bytes if they are padding (less than 8 '\n' bytes). +-- Padding never contains a full section: a section that starts +-- with 0x0a has at least 12 bytes. If the trailing bytes are not padding, we +-- keep the object unchanged and wasm-ld reports the error. +-- +-- In addition, Hadrian passes --format=gnu to emar, thus the archives of the +-- boot libraries have no padding. Cabal does not do this, thus we still need +-- the code above for other libraries and for older installations. +-- +-- References: +-- - llvm-ar formats: https://llvm.org/docs/CommandGuide/llvm-ar.html +-- - Darwin padding and format detection: computeMemberData and +-- NewArchiveMember::detectKindFromObject in +-- llvm/lib/Object/ArchiveWriter.cpp + ------------------------------------------------ ===================================== hadrian/src/Settings/Builders/Ar.hs ===================================== @@ -10,7 +10,11 @@ arBuilderArgs = do stage <- getStage mconcat [ builder (Ar Pack) ? mconcat - [ -- When building on platforms which don't support object merging + [ -- JS target: emar (llvm-ar) uses the Darwin archive format on Darwin + -- hosts. Force the GNU format to avoid padding after wasm objects. + -- See Note [Wasm objects in Darwin archives] in GHC.StgToJS.Object + expr (isJsTarget stage) ? arg "--format=gnu" + , -- When building on platforms which don't support object merging -- we must use the -L flag supported by llvm-ar, which ensures that -- .a files added to the archive are merged into the resulting archive, -- not added as a single file. This requires that we are using llvm-ar ===================================== testsuite/tests/javascript/Makefile ===================================== @@ -31,3 +31,16 @@ T24744: # check that the optimization occurred grep -c "h\$$r1 = h\$$decodeUtf8z" T24744.dump-js grep -c "h\$$stc(h\$$mainZCMainzitestzuvalzu80zulocal,h\$$mainZCMainzitestzuvalzu80zulocal_e,\[h\$$ghczminternalZCGHCziInternalziTypesziunpackCStringzh,h\$$mainZCMainzimain7_1,h\$$mainZCMainzimain7_2\])" T24744.jsexe/out.js + +T25665: + mkdir T25665pkg + '$(TEST_HC)' $(TEST_HC_OPTS) -v0 -this-unit-id t25665-lib -odir T25665pkg -hidir T25665pkg -c T25665Lib.hs + '$(TEST_HC)' $(TEST_HC_OPTS) -v0 -c T25665_js.js -o T25665pkg/T25665_js.o + '$(TEST_HC)' $(TEST_HC_OPTS) -v0 -c T25665_c.c -o T25665_c.o + # add archive padding as llvm-ar does in the Darwin format (--format=darwin) + (cat T25665_c.o; printf '\n\n\n') > T25665pkg/T25665_c.o + '$(AR)' qcS T25665pkg/libHSt25665-lib.a T25665pkg/T25665Lib.o T25665pkg/T25665_js.o T25665pkg/T25665_c.o + '$(GHC_PKG)' init T25665.package.conf + '$(GHC_PKG)' --no-user-package-db -f T25665.package.conf register --force -v0 T25665.pkg + '$(TEST_HC)' $(TEST_HC_OPTS) -v0 -package-db T25665.package.conf -package t25665-lib T25665.hs + ./T25665 ===================================== testsuite/tests/javascript/T25665.hs ===================================== @@ -0,0 +1,11 @@ +-- #25665: link a package whose archive contains a wasm object (built from a C +-- source) followed by archive padding. In the Darwin format (the default of +-- llvm-ar on Darwin hosts), llvm-ar adds '\n' bytes after each member and +-- includes them in the member size. The Makefile adds this padding by hand, +-- thus the test does not depend on llvm-ar. +-- +-- See Note [Wasm objects in Darwin archives] in GHC.StgToJS.Object +import T25665Lib + +main :: IO () +main = print =<< t25665 41 ===================================== testsuite/tests/javascript/T25665.pkg ===================================== @@ -0,0 +1,9 @@ +name: t25665-lib +version: 1.0 +id: t25665-lib +key: t25665-lib +exposed: True +exposed-modules: T25665Lib +import-dirs: ${pkgroot}/T25665pkg +library-dirs: ${pkgroot}/T25665pkg +hs-libraries: HSt25665-lib ===================================== testsuite/tests/javascript/T25665.stdout ===================================== @@ -0,0 +1 @@ +42 ===================================== testsuite/tests/javascript/T25665Lib.hs ===================================== @@ -0,0 +1,3 @@ +module T25665Lib (t25665) where + +foreign import javascript "t25665_wrapper" t25665 :: Int -> IO Int ===================================== testsuite/tests/javascript/T25665_c.c ===================================== @@ -0,0 +1,3 @@ +int t25665_c(int a) { + return a+1; +} ===================================== testsuite/tests/javascript/T25665_js.js ===================================== @@ -0,0 +1,6 @@ +//#OPTIONS:CPP +//#OPTIONS:EMCC:EXPORTED_FUNCTIONS=_t25665_c + +function t25665_wrapper(a) { + return _t25665_c(a); +} ===================================== testsuite/tests/javascript/all.T ===================================== @@ -30,3 +30,4 @@ test('T25633', normal, compile_and_run, ['']) test('T24886', normal, compile, ['']) test('T27788', normal, compile_and_run, ['-O0']) +test('T25665', extra_files(['T25665Lib.hs','T25665_js.js','T25665_c.c','T25665.pkg']), makefile_test, ['T25665']) View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/commit/593f5f5bbe0ab73bbb1cd86e1f8be8f7... -- View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/commit/593f5f5bbe0ab73bbb1cd86e1f8be8f7... You're receiving this email because of your account on gitlab.haskell.org. Manage all notifications: https://gitlab.haskell.org/-/profile/notifications | Help: https://gitlab.haskell.org/help
participants (1)
-
Marge Bot (@marge-bot)