Yes, but this solution is bypassing the problem by using remote execution. My question is what could be done to make the remote execution unnecessary, i.e addressing the "Right now" part. Alan On Sun, Jan 17, 2016 at 4:58 PM, Bartosz Nitka wrote:

From https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi: "We can have multiple instances of a GHC Session on the GHC API, each running TH and/or interpreted code. Right now this is not possible because the linker's state is global."

Regards, Bartosz

2016-01-17 14:49 GMT+00:00 Alan & Kim Zimmerman :

...

At the moment there are issues with having multiple GHC API sessions in a single executable, which boil down to GHC having global variables.

A quick grep over the GHC sources shows the following instances

compiler/ghci/Linker.hs:90:GLOBAL_VAR_M(v_PersistentLinkerState, ... compiler/ghci/Linker.hs:91:GLOBAL_VAR(v_InitLinkerDone, ....

compiler/main/StaticFlags.hs:94:GLOBAL_VAR(v_opt_C, .... compiler/main/StaticFlags.hs:95:GLOBAL_VAR(v_opt_C_ready, ....

compiler/main/DynFlags.hs:4453:GLOBAL_VAR(v_unsafeGlobalDynFlags,....

ghc/GHCi/UI.hs:149:GLOBAL_VAR(macros_ref,....

Of these the v_unsafeGlobalDynFlags can probably be ignored, as they are intended to be used in very specific, limited circumstances only.

So, would it be possible to create a structure housing the remaining ones, and somehow make it possible to access these in some kind of session? Or preferably get rid of them, or move them into existing structures such as DynFlags.

This is an exploratory email, more to get a handle on what has been done/considered already in this space.

The issue does have a major impact on tooling developers though.

Regards Alan _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

I don't think the state-hack tail should wag the dog here. The nub of the problem in practice is the shared linker state isn't it? Simon | -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of Ben | Gamari | Sent: 17 January 2016 19:55 | To: Alan & Kim Zimmerman ; ghc-devs@haskell.org | Subject: Re: Multiple GHC sessions | | Alan & Kim Zimmerman writes: | | > At the moment there are issues with having multiple GHC API sessions | > in a single executable, which boil down to GHC having global | > variables. | > | > A quick grep over the GHC sources shows the following instances | > | > compiler/ghci/Linker.hs:90:GLOBAL_VAR_M(v_PersistentLinkerState, ... | > compiler/ghci/Linker.hs:91:GLOBAL_VAR(v_InitLinkerDone, .... | > | This isn't the only global state had by the linker; have a look at | rts/Linker.c. Unfortunately this would take a fair amount of work to | resolve. The easiest solution here would probably be to put a lock | around the linker and have all sessions in a process use the same | linker. | | > compiler/main/StaticFlags.hs:94:GLOBAL_VAR(v_opt_C, .... | > compiler/main/StaticFlags.hs:95:GLOBAL_VAR(v_opt_C_ready, .... | > | We've been meaning to get rid of the remaining static flags for quite | some time. | | I was a bit surprised to find that opt_NoStateHack is static. This is | a bit of an ugly one: the only user is a test in Id.isStateHackType | which is buried inside several layers [1] of pure predicates that have | no DynFlags. It would be a shame to pass a DynFlags through all of | these calls just to support the state hack (which we threaten drop | about once every six months). | | One option, of course, would be to merge all of the static flags into | DynFlags and rework their users to use unsafeGlobalDynFlags. | | [1] One particular callpath is, | | Id.isStateHackType | Id.typeOneShot | CoreArity.typeArity | CoreArity.exprArity | CorePrep.rhsToBody | CorePrep.cpeBody | | | > | compiler/main/DynFlags.hs:4453:GLOBAL_VAR(v_unsafeGlobalDynFlags,.... | > | The principle user of this is Outputable, which uses it to provide | DynFlags for pprTrace and friends. As you say, these are rather | special cases and it's probably fine if they behave a bit funky in the | case of more than one session in a process. | | > ghc/GHCi/UI.hs:149:GLOBAL_VAR(macros_ref,.... | > | It is completely unclear why this is global at all. It seems like it | would fit just fine in the GHCi monad. I've opened D1789 doing exactly | this. | | There may be other global state that I'm not thinking of, but if this | is everything then it seems quite possible to fix this up for 8.2. | | Cheers, | | - Ben

From a tool writer perspective I would be happy to have GHC/Linker sessions be completely separate from each other. But I guess in practical terms a shared address space needs to be managed, making this impossible.

The external interpreter for GHCI effectively solves the actual loading problem, the other things needing a GHC session are loading a module as far as type checking, for error reporting. I know this gets complicated if there is FFI involved and the loading has to go via object file generation. Alan On Mon, Jan 18, 2016 at 12:53 PM, Ben Gamari wrote:

Simon Peyton Jones writes:

...

I don't think the state-hack tail should wag the dog here. The nub of the problem in practice is the shared linker state isn't it?

I'm not sure; it's not clear to me what would break if the linker were used by more than one session at a time, even with the current GLOBAL_VAR. In fact, it arguably makes a fair bit of sense for all sessions to use the same linker state given that they are also sharing an address space.

The only issue that may require a bit of care is code unloading. Consider that you have two sessions A and B,

1. A loads MyModule 2. B loads MyModule 3. They both use MyModule for a while 4. A unloads MyModule 5. B continues 6. B unloads MyModule

I suspect that things will work fine up to step 6 since we don't actually unload code until the garbage collector has marked the it as dead. That being said, I suspect you would see an error from unloadObj upon attempting to unload the object the second time (and even then, the error is merely a message on stderr). Note, however, that the above conclusions have not been tested.

Otherwise, it appears that things should behave fair well as the PersistentLinkerState is already wrapped in an MVar which should serve to serialize linker calls.

Cheers,

- Ben