Hi, On the road of dynamic linking for Haskell executables, Magnus has pushed a commit [1] on `cblrepo` yesterday. This commit adds the cabal flag `--enable-executable-dynamic`. It follows a previous discussion on this mailing list [2] and solves some of the cited problems. I tried to build `pandoc` dynamically this morning, and there is only one issue left (unless I do something wrong?). For dynamic linking, Cabal/GHC embed an RPATH into the executable: $ readelf --dynamic /usr/bin/pandoc [[...]] 0x000000000000000f (RPATH) Library rpath: [/home/fabien/archlinux/habs/haskell-pandoc/src/pandoc-1.12.1/dist/build:/usr/lib/ghc-7.6.3/site-local/zip-archive-0.1.4:/usr/lib/ghc-7.6.3/site-local/zlib-0.5.4.1:/usr/lib/ghc-7.6.3/site-local/utf8-string-0.3.7:/usr/lib/ghc-7.6.3/site-local/digest-0.0.1.2:/usr/lib/ghc-7.6.3/site-local/binary-0.7.1.0:/usr/lib/ghc-7.6.3/site-local/texmath-0.6.4:/usr/lib/ghc-7.6.3/site-local/xml-1.3.13:/usr/lib/ghc-7.6.3/site-local/temporary-1.1.2.4:/usr/lib/ghc-7.6.3/site-local/tagsoup-0.13:/usr/lib/ghc-7.6.3/site-local/random-1.0.1.1:/usr/lib/ghc-7.6.3/process-1.1.0.2:/usr/lib/ghc-7.6.3/site-local/hslua-0.3.7:/usr/lib/ghc-7.6.3/site-local/data-default-0.5.3:/usr/lib/ghc-7.6.3/site-local/data-default-instances-old-locale-0.0.1:/usr/lib/ghc-7.6.3/site-local/data-default-instances-dlist-0.0.1:/usr/lib/ghc-7.6.3/site-local/data-default-instances-containers-0.0.1:/usr/lib/ghc-7.6.3/site-local/data-default-instances-base-0.0.1:/usr/lib/ghc-7.6.3/site-local/data-default-class-0.0.1:/usr/lib/ghc-7.6.3/site-local/base64-bytestring-1.0.0.1:/usr/lib/ghc-7.6.3/site-local/yaml-0.8.5.1:/usr/lib/ghc-7.6.3/site-local/conduit-1.0.8:/usr/lib/ghc-7.6.3/site-local/void-0.6.1:/usr/lib/ghc-7.6.3/site-local/semigroups-0.11:/usr/lib/ghc-7.6.3/site-local/nats-0.1.2:/usr/lib/ghc-7.6.3/site-local/resourcet-0.4.9:/usr/lib/ghc-7.6.3/site-local/mmorph-1.0.0:/usr/lib/ghc-7.6.3/site-local/lifted-base-0.2.1.0:/usr/lib/ghc-7.6.3/site-local/monad-control-0.3.2.2:/usr/lib/ghc-7.6.3/site-local/transformers-base-0.4.1:/usr/lib/ghc-7.6.3/site-local/base-unicode-symbols-0.2.2.4:/usr/lib/ghc-7.6.3/site-local/pandoc-types-1.12.3:/usr/lib/ghc-7.6.3/site-local/highlighting-kate-0.5.5:/usr/lib/ghc-7.6.3/site-local/pcre-light-0.4:/usr/lib/ghc-7.6.3/site-local/blaze-html-0.6.1.1:/usr/lib/ghc-7.6.3/site-local/blaze-markup-0.5.1.5:/usr/lib/ghc-7.6.3/site-local/extensible-exceptions-0.1.1.4:/usr/lib/ghc-7.6.3/directory-1.2.0.1:/usr/lib/ghc-7.6.3/filepath-1.3.0.1:/usr/lib/ghc-7.6.3/site-local/aeson-0.6.2.1:/usr/lib/ghc-7.6.3/site-local/vector-0.10.9.1:/usr/lib/ghc-7.6.3/site-local/primitive-0.5.1.0:/usr/lib/ghc-7.6.3/site-local/unordered-containers-0.2.3.3:/usr/lib/ghc-7.6.3/template-haskell-2.8.0.0:/usr/lib/ghc-7.6.3/pretty-1.1.1.0:/usr/lib/ghc-7.6.3/site-local/syb-0.4.1:/usr/lib/ghc-7.6.3/site-local/hashable-1.2.1.0:/usr/lib/ghc-7.6.3/site-local/dlist-0.5:/usr/lib/ghc-7.6.3/site-local/blaze-builder-0.3.1.1:/usr/lib/ghc-7.6.3/site-local/attoparsec-0.10.4.0:/usr/lib/ghc-7.6.3/containers-0.5.0.0:/usr/lib/ghc-7.6.3/site-local/HTTP-4000.2.8:/usr/lib/ghc-7.6.3/old-time-1.1.0.1:/usr/lib/ghc-7.6.3/site-local/network-2.4.2.0:/usr/lib/ghc-7.6.3/unix-2.6.0.1:/usr/lib/ghc-7.6.3/time-1.4.0.1:/usr/lib/ghc-7.6.3/old-locale-1.0.0.5:/usr/lib/ghc-7.6.3/site-local/parsec-3.1.3:/usr/lib/ghc-7.6.3/site-local/text-0.11.3.1:/usr/lib/ghc-7.6.3/site-local/mtl-2.1.2:/usr/lib/ghc-7.6.3/site-local/transformers-0.3.0.0:/usr/lib/ghc-7.6.3/bytestring-0.10.0.2:/usr/lib/ghc-7.6.3/deepseq-1.3.0.1:/usr/lib/ghc-7.6.3/array-0.4.0.1:/usr/lib/ghc-7.6.3/base-4.6.0.1:/usr/lib/ghc-7.6.3/integer-gmp-0.5.0.0:/usr/lib/ghc-7.6.3/ghc-prim-0.3.0.0:/usr/lib/ghc-7.6.3] [[...]] This is working to find all libraries except one: ` libHSpandoc-1.12.1-ghc7.6.3.so`. The reason is the following: At compile time, cabal first builds the pandoc library, then the pandoc executable which is using this library. But because the library is not installed at this moment (understand: not located in the final place `/usr/lib/ghc-7.6.3/site-local/pandoc-1.12.1/`) the RPATH embeds its *current* location, in my case `/home/fabien/archlinux/habs/haskell-pandoc/src/pandoc-1.12.1/dist/build` (see the output above). So as long as the library is present in this folder, pandoc works, but obviously that is not working to distribute it. I suppose this method is working for cabal-install because the library stays at its place once built, which is not the case here: the library is moved once built. Also I suppose this problem doesn't appear with library-only package, because each package generate only one `*.so` file (as far as I have seen), so no need to link against another library of the same package at build time. Up to my knowledge I see three solutions: 1. Find a way to build and install in two passes, something like separating in haskell-pandoc and haskell-pandoc-libs (first build and install pandoc-libs, then build and install pandoc) - Requires to find a way to create separate PKGBUILDs for the same hackage package. Elegant but complicated solution, so not very convenient. 2. Try to modify the RPATH inside the executable - Seems to be possible, see [3], but complicated and not very elegant solution. 3. Change the linking paradigm to use /etc/ld.so.conf.d/* instead of the embedded RPATH - Requires to put or link all `*.so` libraries into on place, and add a `/etc/ld.so.conf.d/haskell.conf` file to ghc installation. The most tractable solution imho. Does anyone has remarks/suggestions/ideas/solutions about this problem? Best regards, Fabien [1] https://github.com/magthe/cblrepo/commit/de9df07d398d70255c852f4de53e3d14c4e... [2] http://www.haskell.org/pipermail/arch-haskell/2013-October/004600.html [3] http://stackoverflow.com/questions/13769141/can-i-change-rpath-in-an-already...
On Fri, Nov 01, 2013 at 01:04:44PM +0100, Fabien Dubosson wrote:
Hi,
On the road of dynamic linking for Haskell executables, Magnus has pushed a commit [1] on `cblrepo` yesterday. This commit adds the cabal flag `--enable-executable-dynamic`. It follows a previous discussion on this mailing list [2] and solves some of the cited problems. I tried to build `pandoc` dynamically this morning, and there is only one issue left (unless I do something wrong?).
You did not do something wrong at all. I was a bit to eager pushing the change and seem to have missed in testing the results of that change properly.
This is working to find all libraries except one: ` libHSpandoc-1.12.1-ghc7.6.3.so`. The reason is the following: At compile time, cabal first builds the pandoc library, then the pandoc executable which is using this library. But because the library is not installed at this moment (understand: not located in the final place `/usr/lib/ghc-7.6.3/site-local/pandoc-1.12.1/`) the RPATH embeds its *current* location, in my case `/home/fabien/archlinux/habs/haskell-pandoc/src/pandoc-1.12.1/dist/build` (see the output above). So as long as the library is present in this folder, pandoc works, but obviously that is not working to distribute it. I suppose this method is working for cabal-install because the library stays at its place once built, which is not the case here: the library is moved once built.
Which explains why my crude initial test succeeded, I didn't bother removing the build folder after installation.
Also I suppose this problem doesn't appear with library-only package, because each package generate only one `*.so` file (as far as I have seen), so no need to link against another library of the same package at build time.
That seems logical. It also wouldn't be a problem for exe-only packages since then there is no local lib built.
Up to my knowledge I see three solutions:
1. Find a way to build and install in two passes, something like separating in haskell-pandoc and haskell-pandoc-libs (first build and install pandoc-libs, then build and install pandoc) - Requires to find a way to create separate PKGBUILDs for the same hackage package. Elegant but complicated solution, so not very convenient. 2. Try to modify the RPATH inside the executable - Seems to be possible, see [3], but complicated and not very elegant solution. 3. Change the linking paradigm to use /etc/ld.so.conf.d/* instead of the embedded RPATH - Requires to put or link all `*.so` libraries into on place, and add a `/etc/ld.so.conf.d/haskell.conf` file to ghc installation. The most tractable solution imho.
Does anyone has remarks/suggestions/ideas/solutions about this problem?
Well, to me it seems that Cabal is broken here. On `configure` Cabal is told where the built lib will end up so it would be more correct to put in an RPATH to that location rather than to the build location of the lib. I'll point this out on the haskell-cafe (if no one's beat me to it :). /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: magnus@therning.org jabber: magnus@therning.org twitter: magthe http://therning.org/magnus If voting could really change things it would be illegal.
This patch shows how I propose dealing with Cabal's inability to build
dynamically linked installable executables.
In short:
- Create links to all DSOs from /usr/lib/ghc-7.6.3/shared.
- Rewrite rpath of executables to only contain
/usr/lib/ghc-7.6.3/shared.
Any comments? (Besides the hard coded ghc version in the path ;-)
/M
~~~~
From 6882788bb3df039c9b17a2eef831698030e9cc02 Mon Sep 17 00:00:00 2001
From: Magnus Therning
Any comments? (Besides the hard coded ghc version in the path ;-)
Hum, one question (that may solve the hard coded path ;-) : If the shared libraries need anyway to be in a `shared` folder, why not using a `/etc/ld.so.conf.d/haskell.conf` file (attached to `ghc` package) containing the line `/usr/lib/ghc-7.6.3/shared`, and building *all* Haskell executables/libraries with ghc-option `-dynload=deploy` as explained in [1]? The difference is that executables/libraries will *not* contain a RPATH, but instead will search in LD folders. That will solve the hard coded path and remove the dependency to `chrpath`. Also, from my point of view, this is a more elegant solution since everything will be dynamic and looking into only one place to find libraries. If applying that to libraries doesn't enjoy people (or doesn't work), it is also possible to do that only for executables. But in this later case, building an executable with ghc-option `-dynload=deploy` will probably apply to its library as well. To bypass this problem I think that simply removing the RPATH from the executable will have the same effect. But I haven't tested that yet. PS: If you want to try the LD version, don't forget to run `sudo ldconfig`, after putting the `/etc/ld.so.conf.d/haskell.conf` file. Otherwise the new path will not be taken into account. [1] http://www.haskell.org/ghc/docs/7.6.3/html/users_guide/using-shared-libs.htm... Kind regards, Fabien
On Sun, Nov 10, 2013 at 11:41:18AM +0100, Fabien Dubosson wrote:
Any comments? (Besides the hard coded ghc version in the path ;-)
Hum, one question (that may solve the hard coded path ;-) :
If the shared libraries need anyway to be in a `shared` folder, why not using a `/etc/ld.so.conf.d/haskell.conf` file (attached to `ghc` package) containing the line `/usr/lib/ghc-7.6.3/shared`, and building *all* Haskell executables/libraries with ghc-option `-dynload=deploy` as explained in [1]?
Would you verify that passing --ghc-options when configuring a package adds the options to 'ghc-options:' if present in the .cabal file? /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: magnus@therning.org jabber: magnus@therning.org twitter: magthe http://therning.org/magnus Java is, in many ways, C++--. -- M Feldman
Would you verify that passing --ghc-options when configuring a package adds the options to 'ghc-options:' if present in the .cabal file?
With pleasure. I made a «dummy» program to verify that (files here [1]): 1. A dummy `Main.hs` executable which uses an external library and also generates GHC warnings. 2. A simple `test-ghc-opts.cabal` file which has a `ghc-options:` to display warnings 3. A `build.sh` script which configures the program with a `ghc-options` to remove the [rpath] of dynamic linking. If both the warnings are displayed and the [rpath] doesn't exist, it means that both ways of giving `ghc-options` are working side by side. It is the case on my computer! But I let you verify my scripts to be sure that I'm doing the test correctly. [1] https://gist.github.com/StreakyCobra/0067d15fd74ebf8bf5e0 Kind regards, Fabien
On Sun, Nov 10, 2013 at 10:32:25PM +0100, Fabien Dubosson wrote:
Would you verify that passing --ghc-options when configuring a package adds the options to 'ghc-options:' if present in the .cabal file?
With pleasure. I made a «dummy» program to verify that (files here [1]):
1. A dummy `Main.hs` executable which uses an external library and also generates GHC warnings. 2. A simple `test-ghc-opts.cabal` file which has a `ghc-options:` to display warnings 3. A `build.sh` script which configures the program with a `ghc-options` to remove the [rpath] of dynamic linking.
If both the warnings are displayed and the [rpath] doesn't exist, it means that both ways of giving `ghc-options` are working side by side. It is the case on my computer! But I let you verify my scripts to be sure that I'm doing the test correctly.
[1] https://gist.github.com/StreakyCobra/0067d15fd74ebf8bf5e0
Excellent, then that is indeed a better route than the one I embarked on. /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: magnus@therning.org jabber: magnus@therning.org twitter: magthe http://therning.org/magnus But whereas I previously held for Java a cordial dislike borne of having only a cursory notion of how it worked, now my dislike for the language can no longer be called at all "cordial", for familiarity has bred contempt. -- tom Christiansen
On Mon, Nov 11, 2013 at 06:21:51AM +0100, Magnus Therning wrote:
On Sun, Nov 10, 2013 at 10:32:25PM +0100, Fabien Dubosson wrote:
Would you verify that passing --ghc-options when configuring a package adds the options to 'ghc-options:' if present in the .cabal file?
With pleasure. I made a «dummy» program to verify that (files here [1]):
1. A dummy `Main.hs` executable which uses an external library and also generates GHC warnings. 2. A simple `test-ghc-opts.cabal` file which has a `ghc-options:` to display warnings 3. A `build.sh` script which configures the program with a `ghc-options` to remove the [rpath] of dynamic linking.
If both the warnings are displayed and the [rpath] doesn't exist, it means that both ways of giving `ghc-options` are working side by side. It is the case on my computer! But I let you verify my scripts to be sure that I'm doing the test correctly.
[1] https://gist.github.com/StreakyCobra/0067d15fd74ebf8bf5e0
Excellent, then that is indeed a better route than the one I embarked on.
Another proposal here: http://is.gd/EbUJXu- /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: magnus@therning.org jabber: magnus@therning.org twitter: magthe http://therning.org/magnus As far as the laws of mathematics refer to reality, they are not certain, and as far as they are certain, they do not refer to reality. -- Albert Einstein
Another proposal here: http://is.gd/EbUJXu-
That seems a good solution. I have 2 suggestions: - Remove the `chrpath` dependency which is not needed - Use a ${_ghcversion}="ghc-7.6.3" variable to replace in > text "mkdir ${pkgdir}/usr/lib/ghc-7.6.3/shared" <$> > text "(cd ${pkgdir}/usr/lib/ghc-7.6.3/shared;" <$> > text " for f in $(find .. -name \\*-ghc7.6.3.so); do" <$> But that's just cosmetic :-) I think it is time now to test all this from A to Z! So I would suggest to also create a `dynlinking` branch on the `habs` repository that would contain the GHC's changes (add a `haskell.conf` file, add a call to `ldconfig` in the PKGBUILD). This will let people (or at least us, I don't know if any other person is following this thread ;) the opportunity to test the proposal completely, from the build of GHC to the use of built executables and libraries, with both habs and cblrepo proposed changes. If that succeeds, it will be possible to merge the changes in the main repo (and eventually look further to propose this change to Arch devs that maintains Haskell packages?!?). Kind regards, Fabien
On Sun, Nov 17, 2013 at 09:09:33PM +0100, Fabien Dubosson wrote:
Another proposal here: http://is.gd/EbUJXu-
That seems a good solution. I have 2 suggestions:
- Remove the `chrpath` dependency which is not needed - Use a ${_ghcversion}="ghc-7.6.3" variable to replace in > text "mkdir ${pkgdir}/usr/lib/ghc-7.6.3/shared" <$> > text "(cd ${pkgdir}/usr/lib/ghc-7.6.3/shared;" <$> > text " for f in $(find .. -name \\*-ghc7.6.3.so); do" <$>
But that's just cosmetic :-)
I think it is time now to test all this from A to Z! So I would suggest to also create a `dynlinking` branch on the `habs` repository that would contain the GHC's changes (add a `haskell.conf` file, add a call to `ldconfig` in the PKGBUILD). This will let people (or at least us, I don't know if any other person is following this thread ;) the opportunity to test the proposal completely, from the build of GHC to the use of built executables and libraries, with both habs and cblrepo proposed changes.
I'm currently attempting to re-build all of the packages with these changes in place. You can try them out by adding the following (temporary) repo: [haskell-testing] Server = http://xsounds.org/~haskell/testing/\$arch /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: magnus@therning.org jabber: magnus@therning.org twitter: magthe http://therning.org/magnus For a successful technology, reality must take precedence over public relations, for nature cannot be fooled. -- R.P. Feynman
On Mon, Nov 25, 2013 at 09:27:09PM +0100, Magnus Therning wrote:
On Sun, Nov 17, 2013 at 09:09:33PM +0100, Fabien Dubosson wrote:
Another proposal here: http://is.gd/EbUJXu-
That seems a good solution. I have 2 suggestions:
- Remove the `chrpath` dependency which is not needed - Use a ${_ghcversion}="ghc-7.6.3" variable to replace in > text "mkdir ${pkgdir}/usr/lib/ghc-7.6.3/shared" <$> > text "(cd ${pkgdir}/usr/lib/ghc-7.6.3/shared;" <$> > text " for f in $(find .. -name \\*-ghc7.6.3.so); do" <$>
But that's just cosmetic :-)
I think it is time now to test all this from A to Z! So I would suggest to also create a `dynlinking` branch on the `habs` repository that would contain the GHC's changes (add a `haskell.conf` file, add a call to `ldconfig` in the PKGBUILD). This will let people (or at least us, I don't know if any other person is following this thread ;) the opportunity to test the proposal completely, from the build of GHC to the use of built executables and libraries, with both habs and cblrepo proposed changes.
I'm currently attempting to re-build all of the packages with these changes in place. You can try them out by adding the following (temporary) repo:
[haskell-testing] Server = http://xsounds.org/~haskell/testing/\$arch
I've abandoned this attempt at the moment. I got stuck on building gtk. I haven't looked closely at it, but it seems to build some helper binary as a first step, before building the package itself. This helper refused to link properly with dynamic libs. I don't really have time or patience to conduct such a major rebuild at this point in time. If someone else feels strongly about this and want to spend some time on it, I'll be happy to give any pointers regarding how to get as far as I did. /M -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: magnus@therning.org jabber: magnus@therning.org twitter: magthe http://therning.org/magnus The British have "the perfect temperament to be hackers--technically skilled, slightly disrespectful of authority, and just a touch of criminal behavior". -- Mary Ann Davidson, Oracle's Security Chief
I've abandoned this attempt at the moment. I got stuck on building gtk. I haven't looked closely at it, but it seems to build some helper binary as a first step, before building the package itself. This helper refused to link properly with dynamic libs. I don't really have time or patience to conduct such a major rebuild at this point in time. If someone else feels strongly about this and want to spend some time on it, I'll be happy to give any pointers regarding how to get as far as I did.
Same problem on my side, no free time to spend on this until next month. I will keep this issue in one corner of my mind and try to look at it when I would have more time. Regards, Fabien
participants (2)
-
Fabien Dubosson -
Magnus Therning