[GHC] #11042: Template Haskell / GHCi does not respect extra-lib-dirs

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: #10458 #5289 Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- In particular when compiling on NixOS, but also in many other situations, it is often needed to pass --extra-lib-dirs / --extra-include-dirs to cabal/stack, to help GHC find required system libraries. This works fine for GHC, but not GHCi, and therefore also not Template Haskell code. Minimal example attached. Try it in an environment where libz is not in a standard location, e.g. inside a nix-shell. Note that if you remove the template Haskell splice in Main.hs, the code compiles just fine. This affects GHC 7.10.2, but also HEAD from 2015-08-28, even one including Tamar Christina's 5d841108acef950fed6a5e608ac9b18e7431aa87 commit cherry- picked. I haven't been able to test more recent HEAD, because of unrelated compilation issues. I surmise that a fix to #10458, which currently blocks using HaskellR in GHCi, would also fix this, but I'm not sure. The error message I get is different: {{{ $ stack build --skip-ghc-check --extra-lib- dirs=/nix/store/31w31mc8immhpnmxvcl4l0fvc3i5iwh0-zlib-1.2.8/lib --extra- include- dirs=/nix/store/31w31mc8immhpnmxvcl4l0fvc3i5iwh0-zlib-1.2.8/include b-0.1.0.0: build Preprocessing executable 'b' for b-0.1.0.0... [2 of 2] Compiling Main ( src/Main.hs, .stack- work/dist/x86_64-linux/Cabal-1.23.0.0/build/b/b-tmp/Main.o ) <command line>: can't load .so/.DLL for: libz.so (libz.so: cannot open shared object file: No such file or directory) -- While building package b-0.1.0.0 using: /home/mboes/.stack/setup-exe-cache/setup-Simple- Cabal-1.23.0.0-x86_64-linux-ghc-7.11.20150828 --builddir=.stack- work/dist/x86_64-linux/Cabal-1.23.0.0/ build exe:b --ghc-options " -ddump- hi -ddump-to-file" Process exited with code: ExitFailure 1 }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by mboes): * Attachment "b.tar.gz" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: GHCi crash | Test Case: Blocked By: | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by mboes): * failure: None/Unknown => GHCi crash -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by mboes): * failure: GHCi crash => Compile-time crash -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by Phyx-): Hi @mboes, Thanks for the report. Are you trying to link against a static of dynamic library? The patch in 5d841108acef950fed6a5e608ac9b18e7431aa87 is only for dynamic libraries. I have one locally that I haven't put for review yet for static libraries. I know the error message mentions `libz.so` but that's probably because it gave up and assumed it's an so, which is why I'm wondering if you're trying to link a static lib. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mboes): @Phyx- I'm not sure actually. How do I tell reliably? FWIW, the following command using the above provided example fails too: {{{ $ cabal sandbox init $ cabal install --enable-shared Resolving dependencies... Notice: installing into a sandbox located at /tmp/b/.cabal-sandbox Downloading zlib-0.6.1.1... Configuring zlib-0.6.1.1... Building zlib-0.6.1.1... Installed zlib-0.6.1.1 Configuring b-0.1.0.0... Building b-0.1.0.0... Failed to install b-0.1.0.0 Build log ( /tmp/b/.cabal-sandbox/logs/b-0.1.0.0.log ): Configuring b-0.1.0.0... Building b-0.1.0.0... Preprocessing executable 'b' for b-0.1.0.0... [1 of 2] Compiling Id ( src/Id.hs, dist/dist-sandbox- d8895b0c/build/b/b-tmp/Id.o ) [2 of 2] Compiling Main ( src/Main.hs, dist/dist-sandbox- d8895b0c/build/b/b-tmp/Main.o ) <command line>: can't load .so/.DLL for: libz.so (libz.so: cannot open shared object file: No such file or directory) cabal: Error: some packages failed to install: b-0.1.0.0 failed during the building phase. The exception was: ExitFailure 1 }}} Is there a way to control whether Template Haskell looks for static or for dynamic foreign libraries? Does the --enable-shared flag have any effect on that? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by Phyx-): @mboes, in general by default GHC prefers static libraries, where GHCi prefers dynamic ones. It seems like libz is only available as `libz.a`. So the patch you cherry picked would have no effect. Could you try this branch? https://github.com/Mistuke/ghc/tree/fix-ghci- archive-loading -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by trommler): #10458 has nothing to do with this issue. In #10458 we would like to access a function in a C library specified on the command line. In this ticket GHCi's linker fails to find a shared library that is required by another Haskell package, because the library is installed in a path that is not searched by the operating system's runtime linker. As a workaround you could add the path of `libz.so` to `LD_LIBRARY_PATH`. Suppose, you only want to use Haskell zlib in you Main module, I think GHCi's linker should not try to load `libz.so` at all and merely load the Haskell package's shared library (`libHSzlib<...>.so`). The Haskell shared library contains the appropriate DT_RUNPATH tag pointing to the location of the C library and so it is found and loaded. On ELF systems I would say the fix is to teach `Linker.hs` to not load dependent libraries but rely on the system linker to do the Right Thing. On OS X there are issues with symbols from dependent libraries not being reexported by default. I could look into this once I am done with #10458, but I would rather like to see a document specifying the expected behavior of the GHCi linker and then reimplement it. We also need regression tests for all those special cases! I agree with Austin. The amount of patches starts to smell. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 10458 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by trommler): * blockedby: => 10458 Comment: This will be fixed as part of #10458. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 10458 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mboes): Replying to [comment:7 trommler]:
This will be fixed as part of #10458.
Seeing as #10458 is now closed, does this mean this ticket should be closed too? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by trommler): * blockedby: 10458 => 11238 Comment: Replying to [comment:8 mboes]:
Replying to [comment:7 trommler]:
This will be fixed as part of #10458.
Seeing as #10458 is now closed, does this mean this ticket should be closed too? Unfortunately no.
This ticked should be blocked by #11238 now. In #10458 I originally planned to rework dynamic linking but then we settled for a patch. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by rwbarton): This seems fairly bad and shouldn't block forever on a redesign if there's a way to fix it in the short term. However, I don't understand the issue precisely. The ticket title has a kind of type error, since `extra-lib-dirs` is a flag that is only meaningful to Cabal, not to ghc. Could someone provide a reproducer that does not involve stack or Cabal, or characterize the interaction between Cabal and ghc? This may not even be a ghc bug at all. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

This may not even be a ghc bug at all. I think the ghc bug here is that GHCi panics when it cannot find a shared
#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by trommler): Replying to [comment:10 rwbarton]: [...] library and for errors returned by the link editor and also for some errors returned by `dlopen`. The correct behavior would be to print an error message and terminate normally in such situation as it is not ghc's fault that a file is not found etc. I shall create a separate ticket for that. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

This seems fairly bad and shouldn't block forever on a redesign if
#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by trommler): Replying to [comment:10 rwbarton]: there's a way to fix it in the short term.
However, I don't understand the issue precisely. The ticket title has a
kind of type error, since `extra-lib-dirs` is a flag that is only meaningful to Cabal, not to ghc. Could someone provide a reproducer that does not involve stack or Cabal, or characterize the interaction between Cabal and ghc? Let me describe the issue and perhaps you could help me turn this into a regression test: 1. We have a Haskell package that calls out to a C library. In this ticket the zlib package calls the C library `libz.so`. 1. We want to compile a module that depends on the Haskell package (from 1) and also uses Template Haskell. 1. To compile the module Template Haskell loads all packages that are specified as dependencies (in the Cabal file or on the ghc command line). Currently this means loading all transitive dependencies of each package to be loaded. Package loading fails because `libz.so` cannot be found in any standard directory because on the system @mboes describes it is installed elsewhere. The title of the ticket suggests to specify the directory where `libz.so` is located in the cabal file (of the package using zlib). I think this would violate abstraction: As a client of package zlib I expect zlib to be self-contained and provide all information that is needed to use it. In fact for a dynamically linked zlib package that is indeed the case. When building a dynamically linked package we record an R(UN)PATH for C libraries that are not installed in standard locations. If we just loaded the shared library of package zlib (`libHSzlib<...>.so` we would not need to know where to find `libz.so` because the system runtime loader takes care of that. But currently GHC's dynamic linker mimics the behavior of the (static) RTS linker and would load `libz.so` before it loads `libHSzlib<...>.so`. I could extract a patch from my work on #11238 that implements what I said in the third paragraph of comment:6. It would not only be a short term fix but also be a step in the right direction. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Changes (by trommler): * related: #10458 #5289 => #10458 #5289 #12753 Comment: Replying to [comment:11 trommler]:
Replying to [comment:10 rwbarton]: [...]
This may not even be a ghc bug at all. I think the ghc bug here is that GHCi panics when it cannot find a shared library and for errors returned by the link editor and also for some errors returned by `dlopen`. The correct behavior would be to print an error message and terminate normally in such situation as it is not ghc's fault that a file is not found etc. I shall create a separate ticket for that. Created #12753.
-- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: trommler Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Changes (by trommler): * owner: => trommler -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:14 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: trommler Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Comment (by trommler): Replying to [comment:13 trommler]:
Replying to [comment:11 trommler]:
Replying to [comment:10 rwbarton]: [...]
This may not even be a ghc bug at all. I think the ghc bug here is that GHCi panics when it cannot find a shared library and for errors returned by the link editor and also for some errors returned by `dlopen`. The correct behavior would be to print an error message and terminate normally in such situation as it is not ghc's fault that a file is not found etc. I shall create a separate ticket for that. Created #12753. Wait a minute! In this ticket GHC did not panic but perhaps the error message could be more informative and tell the user why it wanted to load `libz.so` at all.
-- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: trommler Type: bug | Status: closed Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: wontfix | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Changes (by trommler): * status: new => closed * resolution: => wontfix Comment: Replying to [comment:12 trommler]:
Replying to [comment:10 rwbarton]: 3. To compile the module Template Haskell loads all packages that are specified as dependencies (in the Cabal file or on the ghc command line). Currently this means loading all transitive dependencies of each package to be loaded. Package loading fails because `libz.so` cannot be found in any standard directory because on the system @mboes describes it is installed elsewhere. Actually, package zlib should record the location of libz.so if it is not found in a standard location. GHCi reads the `library-dirs` field that should contain the path to `libz.so`.
In changeset:f41a8a369796985a75dd618b969292e1e7033112 a new field `dynamic-library-dirs` has been introduced for cases where dynamic library development files are in a directory separate from the archive (`.a`) files. I checked the code in `Linker.hs` and it reads the `library-dirs` field from the package database. So I would say this is a packaging error on nix. As I said comment:12 passing `extra-lib-dirs` breaks abstraction and is brittle. Closing this as wontfix, please reopen if you disagree. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:16 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: trommler Type: bug | Status: closed Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: wontfix | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Comment (by rwbarton): It may be that `library-dirs` is defined to be the directories in which to search for the Haskell library, not its C dependencies. But then, there is also `ld-options` which could be used to hold `-L` and/or `-rpath` options specifying the location of `zlib`. I agree ''if'' the Haskell library does not have an RPATH pointing at the location of `libz.so`, then ghc is blameless here because (as far as I can tell) it cannot know how to find `libz.so`. Cabal or the end user should be doing something to allow ghc to find the library. If the Haskell library does have an RPATH for `libz.so` then perhaps ghc should not be trying to load the library itself, as you discussed in comment:6. I also agree it's pretty unclear whose responsibility it is supposed to be to do what here... -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Changes (by rwbarton): * status: closed => new * owner: trommler => * resolution: wontfix => Comment:
But currently GHC's dynamic linker mimics the behavior of the (static) RTS linker and would load `libz.so` before it loads `libHSzlib<...>.so`.
Do you know why this is? If we could just not do this, it might fix this issue, right? I'm going to reopen this but mark as infoneeded, though I don't mind much if it ends up being closed again. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:18 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Changes (by rwbarton): * status: new => infoneeded -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Comment (by rwbarton): Maybe related, what is the meaning of this ghc-pkg output: {{{ extra-libraries: z extra-ghci-libraries: }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:20 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Comment (by trommler): Replying to [comment:18 rwbarton]:
But currently GHC's dynamic linker mimics the behavior of the (static) RTS linker and would load `libz.so` before it loads `libHSzlib<...>.so`.
Do you know why this is? In `compiler/ghci/Linker.hs` `linkPackage` all dependencies of a package are linked before we even learn whether the package is in a shared library or an archive. That is probably the reason, but I don't know for sure. If we could just not do this, it might fix this issue, right? Yes, that would fix this issue.
I'm going to reopen this but mark as infoneeded, though I don't mind much if it ends up being closed again. Hmm, should I prepare a patch that implements the fix described here?
-- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Changes (by robinbb): * cc: robinbb (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:22 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Changes (by domenkozar): * cc: domenkozar (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:23 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Changes (by int-index): * cc: int-index (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:24 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Comment (by awson): I've concocted a fix for this based on a very simple idea. The idea is: don't crash early. `load_dyn` simply not crashes when `loadDLL` fails. Instead `load_dyn` issues a warning, stating that it can't load the dll and that some symbols may remain unresolved. If the latter is the case it crashes later eventually. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:25 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Changes (by awson): * Attachment "11042_fix.patch" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Comment (by awson): The fix is a bit partial, but it (at least) fixes the most common case, when we have a regular package built in dyn way. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:26 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Comment (by int-index): Replying to [comment:25 awson]:
The idea is: don't crash early.
Nice! If anyone is willing to test this fix, I've put up a repo with instructions to reproduce this issue: https://github.com/serokell/trac11042 trommler, rwbarton, do you think this is a sensible way forward? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:27 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Comment (by awson): I've amended the patch. Now we don't crash early only if we have a chance to have package's dll loaded, otherwise the linker behaviour is the same as before. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:28 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Changes (by awson): * Attachment "11042_fix.patch" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: (none) Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): #12753 | Wiki Page: | -------------------------------------+------------------------------------- Comment (by awson): https://phabricator.haskell.org/D5170 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:29 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: (none) Type: bug | Status: patch Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Phab:D5170 #12753 | Wiki Page: | -------------------------------------+------------------------------------- Changes (by RyanGlScott): * status: infoneeded => patch * differential: => Phab:D5170 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:30 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: (none) Type: bug | Status: patch Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Phab:D5170 #12753 | Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ericson2314): That patch seems...odd to me. I think we might have a simpler problem: I did on nixos unstable (but I think 18.09 would also work) - clone recentish Cabal - `nix-shell -p 'haskellPackages.ghcWithPackages (p: [ p.zlib ])' -p zlib -p haskellPackages.happy -p haskellPackages.alex` - `new-configure -O0 --enable-tests --extra-lib- dirs=/nix/store/...-zlib-1.2.11/lib` - `cabal new-repl cabal-install` `-L` from `--extra-lib-dirs` did make it to ghci, but it fails with {{{ <command line>: can't load .so/.DLL for: libz.so (libz.so: cannot open shared object file: No such file or directory) }}} and `ghci -v` gives me {{{ *** gcc: /nix/store/10yq7kwlvbc6h658izmrlsspry1g9f3c-gcc-wrapper-7.3.0/bin/cc -fno- stack-protector -DTABLES_NEXT_TO_CODE -B/home/jcericson/.cabal/store/ghc-8.4.3/digest-0.0.1.2-14fe411601bb13fe646394df438bbd46ddfbff1cdb12e63d463c3cc847a007ff/lib --print-file-name libz.so *** gcc: /nix/store/10yq7kwlvbc6h658izmrlsspry1g9f3c-gcc-wrapper-7.3.0/bin/cc -fno- stack-protector -DTABLES_NEXT_TO_CODE -B/home/jcericson/.cabal/store/ghc-8.4.3/digest-0.0.1.2-14fe411601bb13fe646394df438bbd46ddfbff1cdb12e63d463c3cc847a007ff/lib --print-file-name liblibz.so *** gcc: /nix/store/10yq7kwlvbc6h658izmrlsspry1g9f3c-gcc-wrapper-7.3.0/bin/cc -fno- stack-protector -DTABLES_NEXT_TO_CODE -B/home/jcericson/.cabal/store/ghc-8.4.3/digest-0.0.1.2-14fe411601bb13fe646394df438bbd46ddfbff1cdb12e63d463c3cc847a007ff/lib --print-file-name z.lib *** gcc: /nix/store/10yq7kwlvbc6h658izmrlsspry1g9f3c-gcc-wrapper-7.3.0/bin/cc -fno- stack-protector -DTABLES_NEXT_TO_CODE -B/home/jcericson/.cabal/store/ghc-8.4.3/digest-0.0.1.2-14fe411601bb13fe646394df438bbd46ddfbff1cdb12e63d463c3cc847a007ff/lib --print-file-name libz.lib *** gcc: /nix/store/10yq7kwlvbc6h658izmrlsspry1g9f3c-gcc-wrapper-7.3.0/bin/cc -fno- stack-protector -DTABLES_NEXT_TO_CODE -B/home/jcericson/.cabal/store/ghc-8.4.3/digest-0.0.1.2-14fe411601bb13fe646394df438bbd46ddfbff1cdb12e63d463c3cc847a007ff/lib --print-file-name libz.dll.a *** gcc: /nix/store/10yq7kwlvbc6h658izmrlsspry1g9f3c-gcc-wrapper-7.3.0/bin/cc -fno- stack-protector -DTABLES_NEXT_TO_CODE -B/home/jcericson/.cabal/store/ghc-8.4.3/digest-0.0.1.2-14fe411601bb13fe646394df438bbd46ddfbff1cdb12e63d463c3cc847a007ff/lib --print-file-name z.dll.a *** gcc: /nix/store/10yq7kwlvbc6h658izmrlsspry1g9f3c-gcc-wrapper-7.3.0/bin/cc -fno- stack-protector -DTABLES_NEXT_TO_CODE -B/home/jcericson/.cabal/store/ghc-8.4.3/digest-0.0.1.2-14fe411601bb13fe646394df438bbd46ddfbff1cdb12e63d463c3cc847a007ff/lib --print-file-name libz.a *** gcc: /nix/store/10yq7kwlvbc6h658izmrlsspry1g9f3c-gcc-wrapper-7.3.0/bin/cc -fno- stack-protector -DTABLES_NEXT_TO_CODE -B/home/jcericson/.cabal/store/ghc-8.4.3/digest-0.0.1.2-14fe411601bb13fe646394df438bbd46ddfbff1cdb12e63d463c3cc847a007ff/lib --print-file-name z.a Loading package digest-0.0.1.2 ... *** Deleting temp files: Deleting: *** Deleting temp dirs: }}} There's plenty to minimize and control here (e.g. `~/.cabal/store/ghc-8.4.3/package.db` populated in different nix-shells), but I want to ask the simple check, if ghc is looking up libz.so, why isn't it using -L? (Maybe the directories listed in the package db override it?) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:31 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: (none) Type: bug | Status: patch Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Phab:D5170 #12753 | Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ericson2314): Hmm, this may be a cabal my error, but I nuked my `~/.cabal/cache`, did a `cabal new-configure --extra-lib-dir=.*`, and the package conf for `digest` in the new cache still didn't have `library-dirs: ...nix's-zlib`, causing a failure when GHCi attempted to load `digest`. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:32 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: (none) Type: bug | Status: patch Priority: normal | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: 11238 | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Phab:D5170 #12753 | Wiki Page: | -------------------------------------+------------------------------------- Changes (by Ericson2314): * cc: Ericson2314 (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:33 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#11042: Template Haskell / GHCi does not respect extra-lib-dirs
-------------------------------------+-------------------------------------
Reporter: mboes | Owner: (none)
Type: bug | Status: patch
Priority: normal | Milestone:
Component: Compiler | Version: 7.10.2
Resolution: | Keywords:
Operating System: Unknown/Multiple | Architecture:
Type of failure: Compile-time | Unknown/Multiple
crash | Test Case:
Blocked By: 11238 | Blocking:
Related Tickets: #10458 #5289 | Differential Rev(s): Phab:D5170
#12753 |
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Ben Gamari

#11042: Template Haskell / GHCi does not respect extra-lib-dirs -------------------------------------+------------------------------------- Reporter: mboes | Owner: (none) Type: bug | Status: closed Priority: normal | Milestone: 8.8.1 Component: Compiler | Version: 7.10.2 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #10458 #5289 | Differential Rev(s): Phab:D5170 #12753, #11238 | Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: patch => closed * resolution: => fixed * related: #10458 #5289 #12753 => #10458 #5289 #12753, #11238 * blockedby: 11238 => * milestone: => 8.8.1 Comment: As noted in the patch, there isn't quite fixed but it's likely as good as it's going to get. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11042#comment:35 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC