Hi, I'm hacking on shared library support for GHC and it's coming along quite nicely. http://hpaste.org/192 My initial hacks are available from: http://clemens.endorphin.org/patches/ghc-20070605-initial-shared-libs.patch (works only with x86-64 atm, on i386 the NCG dies in the register allocator when compiling cmm files RTS) http://clemens.endorphin.org/patches/cabal-20070528-initial-shared-library.p... libtool usually takes care of creating shared libraries under *nix system. libtool solves a few minor problems associated with: 1) creating shared libraries 2) linking programs that depend on shared libraries 3) running programs that depend on shared libraries libtool is tailored to C compilers and the general opinion from #ghc towards libtool seems to be: "hands off". From the list above, I will try to sketch solutions without libtool. 1) creating shared libraries: At the moment, my second patch teaches Cabal how to build shared libraries. Basically, this is: * add -fPIC to the compiler invocation (and -optc-fPIC for c-sources), * invoke "ld -shared -Bsymbolic -o foo.so obj1.o obj2.o ...". ATM, ld is not invoked with the inter-library dependencies for the shared library being built. This is not problematic as the final executable will include all dependencies due to the ghc package dependency tracking. But DT_NEEDED on ELF influences the sequence in which shared library initializers are run. I have not yet investigated if this leads to any problems. To solve this little shortcoming, the ld-invocation could be delegated to GHC. "ghc -o libHSfoo.so Foo1.o Foo2.o". We already have a similar facility for DLLs (see MkDLL in DriverPipeline.hs). This could be abstracted into MkShared, and platform specific knowledge could be encapsulated in GHC. The benefit would be that we could easily access the package information and we could create shared libraries that contain proper DT_NEEDED sections. 2) Linking programs Linking should work out of the box: "ghc -dynamic -o HelloWorld HelloWorld.o" creates dynamically linked executable. 3) Running programs This a typical problem: ./HelloWorld ./HelloWorld: error while loading shared libraries: libHShaskell98-1.0_dyn.so: cannot open shared object file: No such file or directory There are several ways to add search paths for dynamic linking: either we do it temporarily or we encode the search paths into the executables. On ELF platforms, this works by adding -rpath to the linker flags. This adds two new entries in the .dynamic section (DT_RPATH, DT_RUNPATH) both responsible for signalling additional search paths to the dynamic linker, ld.so. According to Simon Marlow, Windows has similar mechanism via manifest files. Let's see how libtool handles this situation. libtool differentiates between installed and uninstalled libraries. When linking against installed libraries not in the standard search path, libtool uses -rpath to add these search paths to the created executable. When linking against uninstalled libraries, libtool still uses -rpath but pointing to the directory the uninstalled library is going to be installed in. libtool derives this information from the .la files+Makefiles. In any case, libtool creates a wrapper in the build directory that takes care of executing the program linked against uninstalled shared libraries. There are two strategies for accomplishing this: * add the paths of the uninstalled shared libraries to LD_LIBRARY_PATH * relink the executable with additional -rpath's libtool chooses the second strategy. How do we translate these solutions to GHC? The first question is whether we expect ghc -dynamic -package uninstalled-package -o Hello Hello.o ./Hello to work or whether we require manual intervention in these cases. If we expect this to work without intervention, we have the same options as libtool: * create a wrapper that takes care of locating the uninstalled shared libraries and sets LD_LIBRARY_PATH. * create a binary with rpath of the uninstalled libraries, and create an additional executable for deployment without these rpaths. In any case we have to modify the installer scripts to either know about where to locate the real binary, either ask ghc where to find the real binary, or either delegate the installation to ghc. The last option is basically the libtool way, "libtool --mode=install ..." When we decide to create a deployable executable at the "-o" spot, we need to * modify the invocation to manually pick up the libraries by modifying LD_LIBRARY_PATH.. this is pretty unpractical. * delegate invocation to ghc. Maybe "ghc --execute HelloWorld". libtool has a similar mechanism for executing 3rd party programs in the "dynamic environment" of the compiler executable. For instance, "gdb HelloWorld" would fails for libtool as HelloWorld is a wrapper, but "libtool --mode=execute gdb HelloWorld" works, as libtool rewrites to HelloWorld to .libs/lt-HelloWorld. And now something completely different: Create a custom ELF program interpreter for Haskell programs. Using INTERP in the ELF program header, loads up this interpreter and delegates control to it. Usually this is /lib/ld-linux.so.2, the dynamic linker, but we can replace that. Haskell has its own idea of libraries/packages. We have package.conf which gives us the location of the installed libraries. This is ok for static linking, as at link time ghc is running and knows how to invoke gcc with the correct paths. It does not matter, if package.conf is updated afterwards as the statically linked programs contain a copy of the library anyway. For dynamic linking this phase is delayed and when we encode rpath such as "/usr/lib/network-2.0/ghc-6.6/", we can not update to network-2.1 without breaking this executables. A custom programming loading stub could access the global and local package.conf and extract the library path for the dependencies and execve /lib64/ld-linux.so.2 --library-path=<paths of the dependencies> HelloWorld <args> This certainly gives us more flexibility than encoding all these rpaths statically into HelloWorld. To solve the inplace execution directly from the build directory, we might create .HelloWorld.package.conf in case a non-standard package.conf is used (non-standard=different from global and local) and have the stub loader to check for this file. I agree that the last scheme sounds a bit wild, but I argue that that's what ELF designers had in mind when they specified the INTERP header. Of course, this is only a solution for ELF platforms. Opinions :) ? -- Fruhwirth Clemens - http://clemens.endorphin.org