#14444: Linker limit on OS X Sierra breaks builds for big projects -------------------------------------+------------------------------------- Reporter: dredozubov | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 (Linking) | Resolution: | Keywords: Operating System: MacOS X | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): For the record, angerman has suggested in the past that we leverage recursive linking to work around this limitation. He has an example [[https://github.com/angerman/dylib-linking|here]]. However, I'll admit I'm not terribly keen on spending our complexity budget working around arbitrary limitations imposed by Apple. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14444#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14444: Linker limit on OS X Sierra breaks builds for big projects -------------------------------------+------------------------------------- Reporter: dredozubov | Owner: angerman Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 (Linking) | Resolution: | Keywords: Operating System: MacOS X | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by angerman): * cc: angerman (added) * owner: (none) => angerman -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14444#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14444: Linker limit on OS X Sierra breaks builds for big projects -------------------------------------+------------------------------------- Reporter: dredozubov | Owner: angerman Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 (Linking) | Resolution: | Keywords: Operating System: MacOS X | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by chak): * cc: chak (added) Comment:

bgamari: I've been asking people to open tickets with Apple bgamari: As they are really the only ones that can really fix this issue

I am quite sure I know what Apple will respond: "You are holding it wrong." Insider jokes aside, the core problem (as I mentioned before on one of the other tickets) is that GHC tries to use the macOS linker like a Linux linker and it was never meant to be used like that. Hence, Apple will only tell you to use it how it was designed to be used. (This is why I haven't filled a Radar —Apple speak for bug report— against this issue myself.) As @angerman und @shlevy explained, there are alternatives to GHC's current scheme, which are closer to how library paths are usually managed on macOS. In addition, there is a reason why library names on macOS can include directory paths. In combination with @loader_path and @executable_path that always allowed to drastically shorten GHC's load command size. Haskell for Mac has done this for a long time (and I also mentioned this on a previous ticket) — see https://github.com/haskellformac/GHCframework/blob/master/GHCBuild/BuildGHC.... for a messy shell script that is part of how the Haskell for Mac build handles this. (The other parts are in that repo as well.) What I am trying to say here is that Apple will not change this, as they do not consider it a problem caused by macOS, but as a problem caused by GHC's abuse of the MACH-O linker format. Hence, I am afraid, we will have to fix this on our side. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14444#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14444: Linker limit on OS X Sierra breaks builds for big projects -------------------------------------+------------------------------------- Reporter: dredozubov | Owner: angerman Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 (Linking) | Resolution: | Keywords: Operating System: MacOS X | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by dredozubov): FWIW nix people use symlinks to lower the command size https://github.com/NixOS/nixpkgs/blob/a2bebdd6cedb58819cc1738282b9f02964d34a... /haskell-modules/generic-builder.nix#L239 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14444#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14444: Linker limit on OS X Sierra breaks builds for big projects -------------------------------------+------------------------------------- Reporter: dredozubov | Owner: angerman Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 (Linking) | Resolution: | Keywords: Operating System: MacOS X | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by aosivitz): So, I've recently run into this with a large project (using stack). I've tried to make a small reproduce case but have not succeeded. I generated a stack project that builds a library along with 400 dummy dependencies, each of which has a long package name. This is a larger number of dependencies than my original project, but it doesn't trigger the panic. So is there something else I need to do to cause this? Other funny things I've seen: Sometimes the build fails after `slack clean`, but then builds successfully after running it again. The big difference I see is that in the first case the link command is trying to build '/var/folders/.../ghc4371_0/libghc_100.dylib' when it fails, and in the second case it builds '.stack-work/.../libHSpanictest-0.0.0 -8KaC2kNKjFj6cJfkbgZkev-ghc8.2.2.dylib' and succeeds. Is that just a quirk of stack or would it be a meaningful difference? What does it mean to build 'libghc_X.dylib'? I apologize if this info is too stack specific, but it's hard for me to tell where this problem actually comes from. Also, if anyone could advise me on improving my reproduction case that would be very helpful. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14444#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14444: Linker limit on OS X Sierra breaks builds for big projects -------------------------------------+------------------------------------- Reporter: dredozubov | Owner: angerman Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 (Linking) | Resolution: | Keywords: Operating System: MacOS X | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by aosivitz): * cc: aosivitz (added) Comment: Sorry for the delay- I forgot to add myself to CC and didn't see your message. So I have now successfully reproduced the problem in a clean environment with bitemyapp's help. The missing piece is that it fails during template haskell compilation, but I'm a little confused at to why. It makes sense, of course, that template-haskell compilation would involve some dynamic linking in order to run the code, but if the number of dependencies is the problem, doesn't that imply TH links all of its dependencies dynamically? What is the advantage of dynamic linking here instead of just statically linking for TH? Here's a github gist for a shell script that generates a giant stack project (with 150 generated dependencies) that triggers the panic: https://gist.github.com/asivitz/f4b983b2374a6155ac4faaf9b61aca59 I'm not sure the best way to do the same thing without using stack, but if you have an idea I can do that. Thanks! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14444#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14444: Linker limit on OS X Sierra breaks builds for big projects -------------------------------------+------------------------------------- Reporter: dredozubov | Owner: angerman Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 (Linking) | Resolution: | Keywords: Operating System: MacOS X | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari):

Here's another idea: instead of dynamically linking each dep separately for TH codegen, we could combine all the modules into one and then dynamically link in that large blob.

I fear that may increase compilation time rather significantly, however. Linking tends to be quite expensive. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14444#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14444: Linker limit on OS X Sierra breaks builds for big projects -------------------------------------+------------------------------------- Reporter: dredozubov | Owner: angerman Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 (Linking) | Resolution: | Keywords: Operating System: MacOS X | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari):

instead of just loading each dependency directly?

Do you mean directly calling `dlopen` on each dependency? I suppose we could do this and it might even work. The problem is that it does nothing for the case of executables. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14444#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14444: Linker limit on OS X Sierra breaks builds for big projects -------------------------------------+------------------------------------- Reporter: dredozubov | Owner: angerman Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 (Linking) | Resolution: | Keywords: Operating System: MacOS X | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by aosivitz): Oh, I understand. By default GHC will statically link exes, but you can force it to be dynamic and in that case it will fail. Well, I personally am less concerned about that case because I'm not trying to dynamically link any exes, but I see why you would want both to work equally well. It seems to me like the TH problem (building an extra dynamic lib unnecessarily which happens to cause bad behavior on Mac) is separate from the dynamic exe problem (how can we dynamically link a bunch of libs into an exe on Mac). Is there some implementation reason why those two problems are more tied together than I assume? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14444#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14444: Linker limit on OS X Sierra breaks builds for big projects -------------------------------------+------------------------------------- Reporter: dredozubov | Owner: angerman Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 (Linking) | Resolution: | Keywords: Operating System: MacOS X | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by jship): We have worked around this issue by following in the Nix folks' footsteps and using a wrapper script around `ld` to recursively subdivide the dependencies into a tree of re-exporting delegate libraries. The script, an example using aosivitz repro, and more info about this issue is available here: https://github.com/Simspace/ld-wrapper-macos The example in the repo shows how the wrapper script can be used to successfully build a project with 750 direct dependencies. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14444#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14444: Linker limit on OS X Sierra breaks builds for big projects -------------------------------------+------------------------------------- Reporter: dredozubov | Owner: angerman Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 (Linking) | Resolution: | Keywords: Operating System: MacOS X | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by gelisam):

I tried it with a few work projects and it still fails for some of them, which is very frustrating.

Note that the script alters the way in which dylibs are ''created'', not the way in which they are linked. As a consequence, you may have to delete your existing dylib files, otherwise your build tool may choose to keep your old broken dylib files instead of re-creating them using our wrapper script. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14444#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#14444: Linker limit on OS X Sierra breaks builds for big projects -------------------------------------+------------------------------------- Reporter: dredozubov | Owner: angerman Type: bug | Status: patch Priority: normal | Milestone: Component: Compiler | Version: 8.2.1 (Linking) | Resolution: | Keywords: Operating System: MacOS X | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4717 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ben Gamari ): In [changeset:"b592bd98ff25730bbe3c13d6f62a427df8c78e28/ghc" b592bd98/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="b592bd98ff25730bbe3c13d6f62a427df8c78e28" dead strip dylibs on macOS When linking dynamic libraries or executables, we compute the full transitive closure over the dependencies, and instruct the linker to link all dependencies. With deep dependency trees the number of transitive dependencies can grow quickly. macOS since the Sierra release has an upper limit on the load command sizes the linker parses when loading dynamic lirbaries. As such it is mandatory to keep the number of load commands (and their size) small on recent macOS releases. An approach that would just link direct dependencies as specified by the -package-id flag is insufficient, because GHC can inline across packages and the library or executable being linked could refer to symbols deep in the dependency tree. If we just recursively linked librarys and re-exported their symbols, this increases the number of symbols in libraries with many dependencies and ultimately puts excessive strain on the linker to the point where linking takes a lot longer than even the compilation of the modules. We can however build a list of symbols from the obejcts we want to link, and try to compute the libraries we need to link that contain those symbols from the transitive dependency closure. Luckily, we don't need to write this ourselves, but can use the ld64 `-dead_strip_dylibs` linker flag on macOS to achive the same result. This will link only the libraries that are actually referenced, which is usually a small subset of the full transitive dependency closure. As such we should stay within the load command size limit for almost all but pathological cases. Reviewers: bgamari Reviewed By: bgamari Subscribers: lelf, rwbarton, thomie, carter GHC Trac Issues: #14444 Differential Revision: https://phabricator.haskell.org/D4714 }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14444#comment:23 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler