Dynamic libraries and GHCi

We had planned to switch to using a dynamically-linked GHCi for 6.14.1 (see http://hackage.haskell.org/trac/ghc/ticket/3658), which on the whole seems like the right direction to be heading in, since we reduce the dependence on our own RTS linker. The dynamically-linked GHCi works, and passes all the tests. It also starts up faster than the statically linked one, at least on Linux, possibly due to a combination of the system linker being faster than ours and it doing on-demand linking. That's the good news. But I've realised this transition won't be as easy as I thought. Currently if you build a package dynamically (Cabal's --enable-dynamic), then you get a completely separate build of the package, with its own interface files and so on - it's a "way", in the same sense as profiling. This seemed obviously the sensible thing to do, since we really are compiling the code multiple times, and the results might be different. However, it gives us a problem with GHCi. If GHCi is dynamically linked, then it can only load object code that is compiled with -dynamic, because then the object code will be compiled against the right .hi files. I see only three possibilities: - we make -dynamic the default. - we somehow make the two ways compatible. This would probably entail compiling both versions at the same time, so we get the same Core, but running the code generator twice. I think there might be some technical problems with doing this, as we sometimes turn static data into dynamic data when compiling with -dynamic. If we could work around this, though, the general idea would still require a lot of changes in build systems, Cabal, GHC and so on. - packages should have predictable & stable ABIs. I've been wanting to do this for a long time, but we're not ready yet, and even so I hadn't planned to impose stable ABIs on every package - e.g. doing this for the base package would probably limit cross-module optimisation too much. It's perhaps worth investigating the consequences of the first option, making -dynamic the default. Cabal would have to --enable-dynamic by default - perhaps it would get an --enable-static that could even be *off* by default. We would have to measure the performance implications carefully. I know that right now the GC takes quite a severe performance hit (about 30% or so) when compiled with -fPIC on x86, due to losing the %ebx register. I think that's unacceptable, but I haven't managed to fix it yet. I don't have a clear idea for how to proceed, so I thought I'd describe the issues here and see if we can come up with a plan. Cheers, Simon

On Mon, May 10, 2010 at 10:06:21AM +0100, Simon Marlow wrote:
- we make -dynamic the default.
It's perhaps worth investigating the consequences of the first option, making -dynamic the default. Cabal would have to --enable-dynamic by default - perhaps it would get an --enable-static that could even be *off* by default.
I'm a bit confused. When you say "we make -dynamic the default", do you mean "ghc foo.hs" uses the dynamic way? Isn't the important thing that Cabal builds libs the dynamic way by default? We'd still have to build them the static way too, if ghc keeps using static by default. Thanks Ian

On 16/05/2010 20:49, Ian Lynagh wrote:
On Mon, May 10, 2010 at 10:06:21AM +0100, Simon Marlow wrote:
- we make -dynamic the default.
It's perhaps worth investigating the consequences of the first option, making -dynamic the default. Cabal would have to --enable-dynamic by default - perhaps it would get an --enable-static that could even be *off* by default.
I'm a bit confused. When you say "we make -dynamic the default", do you mean "ghc foo.hs" uses the dynamic way?
Yes.
Isn't the important thing that Cabal builds libs the dynamic way by default?
That too. But the point is that if you say $ ghc -c hello.hs $ ghci hello then it works, rather than giving you an obscure error about needing to recompile hello.hs with -dynamic, or perhaps ignoring the .o file completely. A separate issue is that if you say $ cabal install foo $ ghci -package foo then it should work, rather than giving you an obscure error telling you to 'cabal install foo --enable-dynamic --reinstall', or something.
We'd still have to build them the static way too, if ghc keeps using static by default.
Well, perhaps static would not be the default any more. I'm just trying to explore the design space here, at the moment I don't see a solution that is win-win. Cheers, Simon

From: Simon Marlow
We had planned to switch to using a dynamically-linked GHCi for 6.14.1 (see http://hackage.haskell.org/trac/ghc/ticket/3658), which on the whole seems like the right direction to be heading in, since we reduce the dependence on our own RTS linker.
The dynamically-linked GHCi works, and passes all the tests. It also starts up faster than the statically linked one, at least on Linux, possibly due to a combination of the system linker being faster than ours and it doing on-demand linking. That's the good news.
In fact, it's now mandatory for all Mach-O platforms (seemingly including i386-apple-darwin the Tier-1 platform) to completely switch to dynamically-linked GHCi, as those platforms started failing to build static GHCi libraries that are needed by dph packages' use of TemplateHaskell. For patches and details, see: http://hackage.haskell.org/trac/ghc/ticket/3260#comment:5 http://hackage.haskell.org/trac/ghc/ticket/3594#comment:1 But currently there is one problem with "GhcShared=YES": with this option, the stage-2 compiler gets linked dynamically but the corresponding inplace shell wrapper does not set (DY)LD_LIBRARY_PATH, thus ./inplace/bin/ghc-stage2 doesn't run at all. I could work around this by manually symlinking all the dynamic libraries to ./inplace/lib and setting (DY)LD_LIBRARY_PATH to there, but obvisouly there should be a solution better than this. Thanks, PHO _______________________________________________________ - PHO - http://cielonegro.org/ OpenPGP public key: 1024D/1A86EF72 Fpr: 5F3E 5B5F 535C CE27 8254 4D1A 14E7 9CA7 1A86 EF72

On 17/05/2010 04:48, PHO wrote:
From: Simon Marlow
Subject: Dynamic libraries and GHCi Date: Mon, 10 May 2010 10:06:21 +0100 We had planned to switch to using a dynamically-linked GHCi for 6.14.1 (see http://hackage.haskell.org/trac/ghc/ticket/3658), which on the whole seems like the right direction to be heading in, since we reduce the dependence on our own RTS linker.
The dynamically-linked GHCi works, and passes all the tests. It also starts up faster than the statically linked one, at least on Linux, possibly due to a combination of the system linker being faster than ours and it doing on-demand linking. That's the good news.
In fact, it's now mandatory for all Mach-O platforms (seemingly including i386-apple-darwin the Tier-1 platform) to completely switch to dynamically-linked GHCi, as those platforms started failing to build static GHCi libraries that are needed by dph packages' use of TemplateHaskell.
For patches and details, see: http://hackage.haskell.org/trac/ghc/ticket/3260#comment:5 http://hackage.haskell.org/trac/ghc/ticket/3594#comment:1
Ok, but the dynamically-linked GHCi only works by accident right now, because we are loading objects compiled against the static version of the packages but linking them against the dynamic versions. This will result in link errors or crashes eventually.
But currently there is one problem with "GhcShared=YES": with this option, the stage-2 compiler gets linked dynamically but the corresponding inplace shell wrapper does not set (DY)LD_LIBRARY_PATH, thus ./inplace/bin/ghc-stage2 doesn't run at all. I could work around this by manually symlinking all the dynamic libraries to ./inplace/lib and setting (DY)LD_LIBRARY_PATH to there, but obvisouly there should be a solution better than this.
On Linux we link the binary using -rpath (I know OS X doesn't have -rpath). This is another issue we need to resolve before we can switch to a dynamically-linked GHCi. Basically there's a fair bit to do to make a dynamic GHCi a reality, and before we can do anything there are some tricky decisions to make. Cheers, Simon

We had planned to switch to using a dynamically-linked GHCi for 6.14.1
Great! :-)
- we make -dynamic the default. : Cabal would have to --enable-dynamic by default - perhaps it would get an --enable-static that could even be *off* by default.
Yes, I think this is a good approach.. :) Jens
participants (4)
-
Ian Lynagh
-
Jens Petersen
-
PHO
-
Simon Marlow