Hi Alexis,

let me try to provide the high up view. I'm sorry if I'm going a bit overboard on details you already know. But let's

start with clearing up a misconception first. No, GHCi does not always require dynamic linking.

At the very abstract level we have a compiler that knows how to turn various inputs into object code. This includes, C, Cmm, Haskell, assembly (.c, .S, .cmm, .hs -> .o). Thus a complete haskell package ends up as a bunch of object code files. We know that for dynamic linkers, we may need slightly different arguments (e.g. PIC).

We next roll these up into archives (.a) and occasionally a pre-linked object file (e.g. link the whole set of object files into one object file, and resolve internal references); as well as a dynamic (shared object, dylib) file.

For GHCi's purposes (and TH), we ultimately want to call a haskell function to produce some AST to splice in. This haskell

function might not be defined in a different package, but the same, so we'll have to deal with some in-flight packages anyway.

We may habe some Byte Code Object (BCO) glue code to invoke the haskell function, which GHCi will interpret during evaluation. However that function can depend on a large dependency tree, and we don't have BCO for everything. I still think it would be nice to have an abstract machine and an Intermediate Representation/ByteCode, that's a much larger project though. Also until recently BCO's couldn't encode unboxed types/sums even.

So given the BCO glue code, we really want to call into object code (also for performance). You can instruct GHCi to prefer object code as well via -fobject-code.

This now leads us to the need of getting the object code somehow into memory and running it. The dynamic system linker approach would be to turn the object code with the function we want to call into a shared library, and just hand that over to the linker (e.g. dlopen).

However, GHC has for a long time grown it's own in-memory static linker. As such it has the capability to load object file (.o) and resolve them on the fly. There is no need for system shared libraries, a system linker, and to deal with potential bugs in that linker. It also means we can link on platforms that don't have a system linker or a severely restricted one (e.g. iOS).

So from a high level you can look at GHC's RTS linker as a special feature of GHC that allows us to not need a system

provided dynamic linker, if there is none available, or using it is undesirable.

Whether or not stuff is loaded through the internal or external interpreter has near no difference. You _can_ load different abi's through the external iserv (as that iserv can be built against a different abi).

Hope this helps a bit? Feel free to ask more questions.

Cheers,
Moritz

On Wed, 1 Jun 2022 at 03:38, Alexis King <lexi.lambda@gmail.com> wrote:

Hi all,

I’ve recently been trying to better understand how and where time is spent at compile-time when running Template Haskell splices, and one of the areas I’ve been struggling to figure out is the operation of the linker. From reading the source code, here’s a summary of what I think I’ve figured out so far:
TH splices are executed using the GHCi interpreter, though it may be internal or external (if -fexternal-interpreter is used).

Regardless of which mode is used, TH splices need their dependencies loaded into the interpreter context before they can be run. This is handled by the call to loadDecls in hscCompileCoreExpr', which in turn calls loadDependencies in GHC.Linker.Loader.

loadDependencies loads packages and modules in different ways. Package dependencies are just loaded via the appropriate built shared libraries, but modules from the current package have to be loaded a different way, via loadObjects (also in GHC.Linker.Loader).
Here, however, is where I get a bit lost. GHC has two strategies for loading individual objects, which it chooses between depending on whether the current value of interpreterDynamic is True. But I don’t actually understand what interpreterDynamic means! The Haddock comment just says that it determines whether or not the “interpreter uses the Dynamic way”, but I don’t see why that matters. My understanding was that GHCi always requires dynamic linking, since it is, after all, loading code dynamically. Under what circumstances would interpreterDynamic ever be False?

Furthermore, I don’t actually understand precisely how and why this influences the choice of loading strategy. In the case that interpreterDynamic is True, GHC appears to convert the desired dyn_o object into a shared library by calling the system linker, then loads that, which can be very slow but otherwise works. However, when interpreterDynamic is False, it loads the object directly. Both paths eventually call into “the RTS linker”, implemented in rts/Linker.c, to actually load the resulting object.

I have found precious little information on what the RTS linker does, in which contexts it’s used, or how precisely it works. Note [runtime-linker-phases] at the top of Linker.c has some information, but it’s mostly a high-level description of what the code actually does rather than an explanation of its role in the bigger picture. Does anyone know of any resources they could possibly point me to that help to explain how all the pieces fit together here? I’ve spent quite a bit of time reading the code, but I’m afraid I still haven’t managed to see the forest for the trees.

Thanks,
Alexis
_______________________________________________
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs