[GHC] #12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.0.1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: Compile-time Unknown/Multiple | crash Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Hi, git clone (commit dfe55e97ed86567aafca2e5f3c19096e2a4cb50f - Sep 20th 2016) of commercialhaskell.com from github: {{{ https://github.com/commercialhaskell/commercialhaskell.com.git }}} follow instructions from that repo exactly for os x (stacksetup installed ghc-7.10.2, but i have seen similar issue with 8): {{{ $ brew install icu4c Add the following to your ~/.stack/stack.yaml: extra-include-dirs: - /usr/local/opt/icu4c/include extra-lib-dirs: - /usr/local/opt/icu4c/lib Now: $ stack build }}} Hit one minor issue in build process with icu4c, resolving with following stack build: {{{ stack build --extra-lib-dirs=/usr/local/opt/icu4c/lib --extra-include- dirs=/usr/local/opt/icu4c/include }}} reach build of yesod-auth, with a failure: {{{ yesod-auth-1.4.6: configure yesod-auth-1.4.6: build texmath-0.8.2.2: copy/register Progress: 35/38 -- While building package yesod-auth-1.4.6 using: /Users/stephen/.stack/setup-exe-cache/x86_64-osx/setup-Simple- Cabal-1.22.4.0-ghc-7.10.2 --builddir=.stack- work/dist/x86_64-osx/Cabal-1.22.4.0 build --ghc-options " -ddump-hi -ddump-to-file" Process exited with code: ExitFailure 1 Logs have been written to: /Users/stephen/Documents/github/commercialhaskell.com/.stack-work/logs /yesod-auth-1.4.6.log Configuring yesod-auth-1.4.6... Building yesod-auth-1.4.6... Preprocessing library yesod-auth-1.4.6... [ 1 of 11] Compiling Yesod.PasswordStore ( Yesod/PasswordStore.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.4.0/build/Yesod/PasswordStore.o ) /private/var/folders/3k/ycnfbqgx33n7qdytkl9ryx7m0000gn/T/stack49268 /yesod-auth-1.4.6/Yesod/PasswordStore.hs:166:31: Warning: Defaulting the following constraint(s) to type ‘Integer’ (Integral b0) arising from a use of ‘^’ at Yesod/PasswordStore.hs:166:31 (Num b0) arising from the literal ‘32’ at Yesod/PasswordStore.hs:166:32-33 In the first argument of ‘(-)’, namely ‘2 ^ 32’ In the first argument of ‘(*)’, namely ‘(2 ^ 32 - 1)’ In the second argument of ‘(>)’, namely ‘(2 ^ 32 - 1) * hLen’ /private/var/folders/3k/ycnfbqgx33n7qdytkl9ryx7m0000gn/T/stack49268 /yesod-auth-1.4.6/Yesod/PasswordStore.hs:419:1: Warning: Defined but not used: ‘toStrict’ /private/var/folders/3k/ycnfbqgx33n7qdytkl9ryx7m0000gn/T/stack49268 /yesod-auth-1.4.6/Yesod/PasswordStore.hs:422:1: Warning: Defined but not used: ‘fromStrict’ [ 2 of 11] Compiling Yesod.Auth.Message ( Yesod/Auth/Message.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.4.0/build/Yesod/Auth/Message.o ) /private/var/folders/3k/ycnfbqgx33n7qdytkl9ryx7m0000gn/T/stack49268 /yesod-auth-1.4.6/Yesod/Auth/Message.hs:22:1: Warning: The import of ‘Data.Monoid’ is redundant except perhaps to import instances from ‘Data.Monoid’ To import instances alone, use: import Data.Monoid() [ 3 of 11] Compiling Yesod.Auth.Routes ( Yesod/Auth/Routes.hs, .stack- work/dist/x86_64-osx/Cabal-1.22.4.0/build/Yesod/Auth/Routes.o ) [ 4 of 11] Compiling Yesod.Auth ( Yesod/Auth.hs, .stack- work/dist/x86_64-osx/Cabal-1.22.4.0/build/Yesod/Auth.o ) ghc: panic! (the 'impossible' happened) (GHC version 7.10.2 for x86_64-apple-darwin): Loading temp shared object failed: dlopen(/var/folders/3k/ycnfbqgx33n7qdytkl9ryx7m0000gn/T/ghc64990_0/libghc_21.dylib, 5): no suitable image found. Did find: /var/folders/3k/ycnfbqgx33n7qdytkl9ryx7m0000gn/T/ghc64990_0/libghc_21.dylib: malformed mach-o: load commands size (34176) > 32768 Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug }}} So reporting.... I've seen the same error attempting to brew install haskell-stack on sierra betas 3 and 4, but on those occasions, an issue with libghc_44.dylib causing the problem. {{{ [ 4 of 87] Compiling System.Process.Read ( src/System/Process/Read.hs, dist/dist-sandbox-558713ad/build/System/Process/Read.o ) ghc: panic! (the 'impossible' happened) (GHC version 8.0.1 for x86_64-apple-darwin): Loading temp shared object failed: dlopen(/var/folders/3k/ycnfbqgx33n7qdytkl9ryx7m0000gn/T/ghc67839_0/libghc_44.dylib, 5): no suitable image found. Did find: /var/folders/3k/ycnfbqgx33n7qdytkl9ryx7m0000gn/T/ghc67839_0/libghc_44.dylib: malformed mach-o: load commands size (40560) > 32768 Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug }}} stack version: {{{ Version 1.1.2, Git revision cebe10e845fed4420b6224d97dcabf20477bbd4b (3646 commits) x86_64 hpack-0.14.0 }}} stack exec env returns: {{{ stack exec env Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.tEAdTiAWFI/Render GHC_PACKAGE_PATH=/Users/stephen/Documents/github/commercialhaskell.com /.stack- work/install/x86_64-osx/lts-3.0/7.10.2/pkgdb:/Users/stephen/.stack/snapshots/x86_64-osx/lts-3.0/7.10.2/pkgdb:/Users/stephen/.stack/programs/x86_64-osx/ghc-7.10.2/lib/ghc-7.10.2/package.conf.d HASKELL_DIST_DIR=.stack-work/dist/x86_64-osx/Cabal-1.22.4.0 HASKELL_PACKAGE_SANDBOX=/Users/stephen/.stack/snapshots/x86_64-osx/lts-3.0/7.10.2/pkgdb HASKELL_PACKAGE_SANDBOXES=/Users/stephen/Documents/github/commercialhaskell.com /.stack- work/install/x86_64-osx/lts-3.0/7.10.2/pkgdb:/Users/stephen/.stack/snapshots/x86_64-osx/lts-3.0/7.10.2/pkgdb: HOME=/Users/stephen LANG=en_IE.UTF-8 LOGNAME=stephen NAME=Stephen Barrett OLDPWD=/Users/stephen/Documents/github PATH=/Users/stephen/Documents/github/commercialhaskell.com/.stack- work/install/x86_64-osx/lts-3.0/7.10.2/bin:/Users/stephen/.stack/snapshots/x86_64-osx/lts-3.0/7.10.2/bin:/Users/stephen/.stack/programs/x86_64-osx/ghc-7.10.2/bin:/Users/stephen/Library/Haskell/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin PWD=/Users/stephen/Documents/github/commercialhaskell.com SECURITYSESSIONID=186a6 SHELL=/bin/bash SHLVL=1 SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.RTM6V8Lo2y/Listeners STACK_EXE=/Library/Haskell/ghc-8.0.1-x86_64/bin/stack TERM=xterm-256color TERM_PROGRAM=Apple_Terminal TERM_PROGRAM_VERSION=377 TERM_SESSION_ID=020844C9-1E8A-410E-8892-43DC0C5A8C0B TMPDIR=/var/folders/3k/ycnfbqgx33n7qdytkl9ryx7m0000gn/T/ USER=stephen XPC_FLAGS=0x0 XPC_SERVICE_NAME=0 _=/usr/local/bin/stack __CF_USER_TEXT_ENCODING=0x1F5:0x0:0x6C }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mpickering): Stephen, thanks for the instructions but it would be much easier to make progress if they were simpler. * Didn't rely on stack (essential) * Was reproducible with the latest GHC version * Didn't depend on lots of packages. Could you try removing code and packages/dependencies from `yesod-auth` so that we can a standalone reproducible case? Ideally the reproduction would just be a single command line invocation of GHC. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => infoneeded Comment: Is it possible that your `libghc` is built with `-split-sections`? It sounds like we are producing too many sections for Darwin's linker. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by thomie): * os: Unknown/Multiple => MacOS X * related: => #12198 Comment: Also reported as #12198. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mistydemeo): I reported #12198, which happened when the ghc being used to build As far as I'm aware, `-split-sections` is off by default, and the Homebrew buildscript doesn't explicitly enable it: https://github.com/Homebrew /homebrew- core/blob/df621e49f7006c9aa7ed7d1f570c955447f160a8/Formula/ghc.rb I don't believe stack is building with it either. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Can someone please provide the output of `objdump -x` (or Darwin's equivalent) from the `libghc` in question? I really don't know what could be producing tens of thousands of load commands if not `-split-sections`. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mistydemeo): GHC is deleting the temp shared object immediately after the build fails; is there a flag I can set to keep it? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mistydemeo): Okay, got it - I fetched the actual `ghc` invocation from `cabal` and added `-keep-tmp-files`. I've attached the output of `otool -l`, which contains the load commands, and the dylib itself. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by mistydemeo): * Attachment "load_commands.txt" added. Output of otool -l libghc_29.dylib -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by mistydemeo): * Attachment "objdump_x.txt" added. Output of gobjdump -x (GNU objdump, not LLVM objdump) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by mistydemeo): * Attachment "libghc_29.dylib.xz" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Carter astutely pointed out that `config.mk` enables GHC's `SplitObjs` build system setting on those platforms that support it (of which OS X is one). The real question is why this only started breaking now. I checked the Mach-O specification and there is no mention of any limit on the number of load commands, so presumably this is some artificial limit imposed by the linker. Carter suggested that perhaps Sierra moved to the LLVM linker, `lld`, but I see no mention of this error message in the `lld` repository. Very mysterious. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * milestone: => 8.0.2 Comment: Actually, it looks like we have little choice but to disable split sections on all OS X builds since we won't know what linker is being used to do the final link until runtime. Sadly this means that Darwin binary distribution sizes will grow but that's a price that OS X users will just need to pay. I'll get a patch in to 8.0.2 doing this. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): As it turns out there is already autoconf logic to disable to detect related Apple brokenness in `configure.ac` (#4013). That being said, it seems quite insufficient since, as related above, there's no reason to believe that the linker at `configure`-time is the same linker that will be used by the built compiler. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * differential: => Phab:D2532 Comment: I think we want something like Phab:D2532 although I'm still investigating. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Austin says, {{{ <thoughtpolice> bgamari: FWIW, the load commands stuff is part of dyld, not LLVM at all. Which makes sense because this is a dynamic linking problem at load time, and not one with the object linker trying to statically link some objects together at runtime ("Loading temp shared object failed..." implies it actually built the dylib). <thoughtpolice> The bad part about this is Apple hasn't published the new source code for dyld in Sierra, so there's no rhyme or reason as to why just yet. Look at 'ImageLoaderMachO::sniffLoadCommands' here: http://opensource.apple.com/source/dyld/dyld-353.2.1/src/ImageLoaderMachO.cp... }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by rwbarton): We did some investigation last night in IRC and came to the conclusion that the runtime loader in Sierra has a new limit on the `sizeofcmds` field which is the total size of all the load commands in the dynamic library. Looking at the `otool -l` output in an earlier attachment, we can see that the main contributors to the size are primarily the RPATH entries and to a lesser extent the LOAD_DYLIB entries. There are over 100 of each, and the paths in the RPATH entries are quite long (I guess this is why stack is affected in particular). So, I don't expect that disabling split objects will help at all, unfortunately. There seems to just be a limit on the total size of RPATH entries we can use. In fact Cabal is the one choosing these RPATHs, not GHC itself. So this will require some kind of change to Cabal to decrease the total size of the RPATH entries it produces. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by rwbarton): Actually GHC also sets RPATHs on the temporary dynamic libraries it builds (see `linkDynLib`), which is the instance in this ticket. So both GHC and Cabal will need some kind of workaround. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:14 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by jonathan): Just for reference, I built GHC with `split-sections` disabled with no luck as previously expected. As @rwbarton mentioned, this occurs when there are a large number of dependencies which overflow past the RPATH limits (building Stack itself is a good test). Has a ticket been created for Cabal yet? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by rwbarton): The OS X linker has an option '''-dylib_file''' ''install_name:file_name'' Specifies that a dynamic shared library is in a different location than its standard location. Use this option when you link with a library that is dependent on a dynamic library, and the dynamic library is in a location other than its default location. ''install_name'' specifies the path where the library normally resides. ''file_name'' specifies the path of the library you want to use instead. For example, if you link to a library that depends upon the dynamic library libsys and you have libsys installed in a nondefault location, you would use this option: '''-dylib_file /lib/libsys_s.A.dylib:/me/lib/libsys_s.A.dylib'''. So GHC could do something like this. When it needs to link a temporary shared library against a Haskell package, rather than using `-Ll -Wl,-rpath -Wl,l` to add the package's library directory `l` to the rpath, since `l` typically has a form like `/home/rwbarton/.cabal/lib/x86_64 -linux-ghc-7.8.4/text-1.1.1.3`, instead add the parent directory of `l` to the rpath and use `-dylib_file` to link against the library with an install name of `@rpath/text-1.1.1.3/libHStext-1.1.1.3-ghc7.8.4.dylib`. Since the parent directories of the package libraries will generally be mostly the same (like `/home/rwbarton/.cabal/lib/x86_64-linux-ghc-7.8.4`), they can be deduplicated to save most of the size of the load commands. This is kind of a hack that relies on the directory structure that Cabal produces to be effective, but it has the advantage of being a local change. Cabal would also need some similar change in order for the final libraries that it links to be loadable. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:16 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Perhaps this would be a good excuse to address #11587. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by rwbarton): Absolutely, but it might be tough to do on short notice; I'm guessing it requires a coordinated change to Cabal, and might break other build systems. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:18 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Changes (by glguy): * cc: glguy (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

This is kind of a hack that relies on the directory structure that Cabal
#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by duncan): produces to be effective, but it has the advantage of being a local change. Cabal would also need some similar change in order for the final libraries that it links to be loadable. Unfortunately, in general it is the user/builder calling Cabal that gets to choose the layout, so it's hard to pick a fixed scheme. That said, we can in principle compute an optimal (or near-optimal) set of shared prefixes. We start with the info from the ghc-pkg db (InstalledPackageInfo) which has the full path to the libraries (or enough info to compute it). Then we would take the set of those paths, make a trie, count the number of elements beneath each trie node and take as shared prefixes all the ones with a count of > 1. Or something like that. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:20 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by rwbarton): I suppose another dumb hack would be to just create symlinks to all our dependencies in the GHC temporary directory and set that as the RPATH... -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

We did some investigation last night in IRC and came to the conclusion
#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by chak): Replying to [comment:13 rwbarton]: that the runtime loader in Sierra has a new limit on the `sizeofcmds` field which is the total size of all the load commands in the dynamic library. Looking at the `otool -l` output in an earlier attachment, we can see that the main contributors to the size are primarily the RPATH entries and to a lesser extent the LOAD_DYLIB entries. There are over 100 of each, and the paths in the RPATH entries are quite long (I guess this is why stack is affected in particular).
So, I don't expect that disabling split objects will help at all,
unfortunately. There seems to just be a limit on the total size of RPATH entries we can use.
In fact Cabal is the one choosing these RPATHs, not GHC itself. So this
will require some kind of change to Cabal to decrease the total size of the RPATH entries it produces. I suspect that this particular use of RPATHs is a remnant of trying to deal with dylibs on macOS as its done on Linux. On macOS, there is a pretty simple fix to this (which is what I use in Haskell for Mac btw). The library name of a dylib in macOS can include a path component. So, instead of just using the filename of a library by itself as its install name, simply use the filename '''prefixed''' by the package directory. So, instead of `base-x.y.z-ABCD.dylib` use `base-x.y.z/base-x.y.z-ABCD.dylib`. Then, only one RPATH is needed, namely the GHC LIBDIR (or wherever the package db is). This keeps the load command size down and solves the problem in #11587. (Incidentally, this approach in combination with using `@loader_path` in RPATH specs also allows to make relocate package dbs, which is important if you want to ship a macOS app that includes dynamically linked Haskell code.) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:22 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Changes (by chak): * cc: chak (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:23 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by darchon): Given that I'm the one who wrote GHC's {{{@rpath}}}-relative {{{install_name}}} code, and Cabal's {{{RPATH}}}-handling code, I want to point out one problem that needs some thought before we make all the {{{install_name}}}s {{{@rpath/libname-x.y.z/libname-x.y.z.dylib}}}: sometimes, the libraries that we link against aren't in their installed location yet. And there is one particular case when this is a problem: when a Cabal {{{testsuite}}} executable is dynamically linked against the library under development. In this case, the {{{libname-x.y.z.dylib}}} is still in {{{./dist/build}}}. Now, because (currently) the {{{install_name}}} is simply {{{@rpath/libname-x.y.z.dylib}}}, we can simply execute the {{{testsuite}}} executable where the {{{DYLD_LIBRARY_PATH}}} contains {{{./dist/build}}}. If the {{{install_name}}} is going to be changed to {{{@rpath/libname-x.y.z/libname-x.y.z.dylib}}}[1], then we have to take care of the above-mentioned problem. [1] Another solution is of course to put (or symlink) all the {{{.dylib}}}s in the {{{$lib}}} directory instead of {{{$lib/libname-x.y.z}}}, which I've seen being suggested, but is something I personally don't like it. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:24 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by darchon): Also, I'll be attending the hackathon on 8 and 9 October (after Haskell Exchange 2016), so if this problem isn't already fixed by that time, I volunteer to work on a patch during the hackathon. I won't have time to work on it on any other date or time. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:25 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: normal | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Changes (by mnislaih): * cc: mnislaih (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:26 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Changes (by simonpj): * priority: normal => highest Comment: Gershom thinks that shipping GHC 8.0.2 that does not work on OSX's latest release would be worse than delaying 8.0.2. So I'm making priority = highest. (Suggestion only of course.) Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:27 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by rwbarton): So I think there are three places in total where RPATHs are chosen. * In GHC, when building a temporary shared library to load the current package for ghci or TH. * In GHC, when doing the final link of a dynamic executable or library using `-dynload system` (the default) * In Cabal, when doing the final link of a dynamic executable or library (invoking GHC with `-dynload deploy` and explicit `-optl` options to set the RPATH) The first issue is essentially local, since the temporary shared library is needed just once. But the other two issues are not, and changing the way we set up the RPATHs and install names has side effects, for example in the case darchon mentioned above, or for someone who deploys their application with all the Haskell libraries in a single directory. So it's a bit of a sticky situation. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:28 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: infoneeded Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by gershomb): I'm not sure how the triage system works these days. Should this be removed from infoneeded to show up in the standard lists at this point of tickets with status new? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:29 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: infoneeded => new Comment: Indeed it should be in `new`. Thanks for pointing this out, Gershom. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:30 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by carter): Just to be clear, the ghc work flows that are impacted are those which need the system linker. Namely any dynamic linked executables , ghci , or TH. I agree with Gershom's assessment that 8.0.2 should wait till this is validated as being fixed for 10.12 and at least doesn't trigger any regressions on 10.11 and 10.10. @ilovezfs will you be able to help us consistently test that version range for correctness and absence of regressions? I could probably help with some guidance. I guess this is the first time Mac OS X support range has come up since the cpp / clang challenges in recent years. Or the more recent issue of some object code tool from llvm, nm, not defaulting to posix format for certain outputs. I would propose that we treat at least the 2-3 most recent OS X releases as "core tier 1" of our tier 1 Apple support. At least as an informal guide post? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:31 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Changes (by chak): * cc: chak (removed) * cc: chak@… (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:32 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): To be clear, while I agree that this is an important issue, 8.0.2 needs to leave the dock at some point soon as 8.2.1 is on the horizon. I'm okay with pushing the 8.0.2 release off by up to two weeks for this, but at that point I will really need to release it regardless of whether this issue is fixed. I am working with Oleg to get our OS X test box running Sierra, but I can't make any promises about being able to fix this myself. It would be quite helpful if someone could summarize the proposed solution. Even better, it would be amazing if someone with an affected machine and knowledge of dynamic linking on OS X (a small set, I know!) could take ownership of this issue and carry it to solution. This may be darchon, but I'm a bit worried what might happen if he doesn't have time to finish, October 10th arrives, and we have no solution. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:33 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari):
I would propose that we treat at least the 2-3 most recent OS X releases as "core tier 1" of our tier 1 Apple support. At least as an informal guide post?
I'd say we really can't do that unless we have the ability to readily test these releases. In principle [[https://ghc.haskell.org/trac/ghc/wiki/Platforms#Tier1platforms|Tier 1 platforms]] are supposed to have a sponsor and some sort of regular testing. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:34 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by carter): @darchon could you share any further Brain dump / background / design references either here and or shoot me an email? I'm willing to carve out some work time this week to see if I can help get the ball rolling, but I'd need at least a brain dump or light guidance. I'm not exactly familiar with linker shenanigans but I can probably get at least a skeleton of a patch together if you could orient me towards what pieces of ghc/cabal need tweaking and such. But i need a teeny bit of guidance to make sure it's a good jump start for you or such -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:35 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Changes (by dleuschner): * cc: dleuschner (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:36 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by darchon): @carter, braindump as requested: * After reading @rwbarton's comment about a person wanting to pack up `dynlib`s for deployment in a single directory, I think I've changed my mind about installation layout. That is, I think the `install_name` of dynlibs should stay `@rpath/libname-x.y.z.dylib` as they are now. * This means that, upon installation, dynlibs should be copied to `$libdir/libname-x.y.z.dylib`, instead of the current `$libdir/libname-x.y.z/libname-x.y.z.dyblib`. * The `.hi` files should still be copied to the `$libdir/libname-x.y.z/` directory. * Perhaps the `a` static lib should also be copied to `$libdir/libname-x.y.z.ar`, for consistency with the `dynlib` dynamic lib. * Due to the use of `nub`, Cabal (in the case of `dynload deploy`) will then already make sure we only end up with a couple of `RPATH` equal to the `$libdir`s of the installed packages. * I'm not sure, but I think GHC also `nub`s the `RPATH`s (in the case of `dynload system`), this would have to be checked. * This plan doesn't involve mucking about with dynamic linking commands. So, if we go with the "put-all-the-dynlibs-in-single-directory" plan, which is what I now prefer, then we would need to do the following: * Add a `--libifacedir` setting to `Cabal`, which is, like `--libsubdir` implicitly prefixed by `$libdir`. `--libifacedir` determines where the `.hi` files go, and the default should be the same as `--libsubdir` (for all platforms) * Change the purpose of `--libsubdir` to mean the the location of the `.dylib`/`.so`/`.a` files. * For OSX only, change the default `libsubdir` to `.`, i.e. equal to `$libdir`. * Update `GHC`s Makefile, so that, on OS X, `make install` puts the `.dylib`/`.so`/`.a` in `$libdir`. Things that still need figuring out: * Do the above changes mean changes in the package database format? if so, what? * Do we need to update `stack` to use the new `--libifacedir` as well? or this it simply take over the default Cabal settings? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:37 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by carter): Let the record show that darchon is ChristianB :) I'll ask you some more off line so I can better understand -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:38 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by carter): I'm not sure how much headway I'll make between now and the weekend, what pieces of this plan are / are not obvious? One issue that came up in the IRC chat with Duncan is cabal new build style needs treatment too? What are the low hanging action items , the easy ones, and the tricky bits? I have Sierra hardware now, so there's that At the very least I'll make sure I file a radar this week -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:39 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): darchon said,
I'm not sure, but I think GHC also nubs the `RPATH`s (in the case of dynload system), this would have to be checked.
For the record, I believe we compute the linker command line in `SysTools.linkDynLib`, which calls `collectLibraryPaths` to compute the set of needed `RPATH` directories. `collectLibraryPaths` indeed calls `nub`, but it does **not** `sort` before doing so, meaning that `nub` won't always remove all duplicates. This should be fixed. `Cabal` should also be checked for this issue. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:40 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Oops, please excuse the cognitive malfunction above; `nub` does not require sorted input. It seems GHC is fine with respect to `RPATH` deduplication. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:41 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): darchon said,
* For OSX only, change the default `--libsubdir` to `.`, i.e. equal to `$libdir`. * Update GHCs `Makefile`, so that, on OS X, `make install` puts the .dylib/.so/.a in `$libdir`.
I'm not sure we want to limit this to OS X. Having many `RPATH` directories is also a startup performance issue on Linux (and likely other platforms) since the dynamic linker does a linear search through every `RPATH` directory for every library it needs to load. This was noticed in #11587. The solution to this is to move all dynamic libraries into `$libdir`, as you suggest. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:42 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: new Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by darchon): Once https://github.com/haskell/cabal/pull/3955 is merged, and a new point-release of Cabal-v1.24 is made, I think this issue will be solved by including the new point release in GHC-8.0.2. I did a {{{cabal install stack --enable-executable-dynamic}}} on OS X Sierra, and the build succeeded, and the resulting {{{stack}}} executable worked as expected. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:43 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: upstream Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => upstream -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:44 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: upstream Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Changes (by lelf): * cc: lelf (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:45 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra beta 4 -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: upstream Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Changes (by tohnann): * cc: tohnann (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:46 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: upstream Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:47 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: upstream Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by ilovezfs): There's a proposal for a partial fix in https://github.com/haskell/cabal/pull/3982 (and backported to Cabal 1.24 in https://github.com/haskell/cabal/pull/3983). "Partial" fix in that the problem persists unaddressed for `cabal new- buld`, which will still trigger the "malformed mach-o: load commands size" failure. I have used the PR to build git-annex in a cabal sandbox successfully (hurray). Attempting to do the same with cabal new-build still fails as "expected." -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:48 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra
-------------------------------------+-------------------------------------
Reporter: stephenb | Owner:
Type: bug | Status: upstream
Priority: highest | Milestone: 8.0.2
Component: Compiler | Version: 8.0.1
Resolution: | Keywords:
Operating System: MacOS X | Architecture:
Type of failure: Compile-time | Unknown/Multiple
crash | Test Case:
Blocked By: | Blocking:
Related Tickets: #12198 | Differential Rev(s): Phab:D2532
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Edward Z. Yang

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: closed Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: upstream => closed * resolution: => fixed Comment: Merged to `ghc-8.0` as 2722cd55431452ad7fc7fcae030b1587086dec8f. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:50 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra
-------------------------------------+-------------------------------------
Reporter: stephenb | Owner:
Type: bug | Status: closed
Priority: highest | Milestone: 8.0.2
Component: Compiler | Version: 8.0.1
Resolution: fixed | Keywords:
Operating System: MacOS X | Architecture:
Type of failure: Compile-time | Unknown/Multiple
crash | Test Case:
Blocked By: | Blocking:
Related Tickets: #12198 | Differential Rev(s): Phab:D2532
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Ben Gamari

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: closed Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): comment:51 merged to `ghc-8.0` as e7017ca8b02ad7abb7c6cd9100e44fb8a215985f. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:52 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra -------------------------------------+------------------------------------- Reporter: stephenb | Owner: Type: bug | Status: closed Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by ilovezfs): Sources for Sierra's dyld have now been posted: https://opensource.apple.com/source/dyld/dyld-421.1/ https://opensource.apple.com/tarballs/dyld/dyld-421.1.tar.gz https://opensource.apple.com/source/dyld/dyld-421.1/src/ImageLoader.h.auto.h... {{{ #define MAX_MACH_O_HEADER_AND_LOAD_COMMANDS_SIZE (32*1024) }}} https://opensource.apple.com/source/dyld/dyld-421.1/src/ImageLoaderMachO.cpp... {{{ if ( sizeofcmds > (MAX_MACH_O_HEADER_AND_LOAD_COMMANDS_SIZE- sizeof(macho_header)) ) dyld::throwf("malformed mach-o: load commands size (%u) > %u", sizeofcmds, MAX_MACH_O_HEADER_AND_LOAD_COMMANDS_SIZE); }}} Note the changes from El Capitan https://opensource.apple.com/source/dyld/dyld-360.22/src/ImageLoaderMachO.cp... Other Sierra sources: https://opensource.apple.com/release/os-x-1012.html -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:53 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra -------------------------------------+------------------------------------- Reporter: stephenb | Owner: (none) Type: bug | Status: closed Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by dredozubov): I'm experiencing the same problem with OS X Sierra: {{{ [ 53 of 176] Compiling Auth.DB.Model.User ( src/Auth/DB/Model/User.hs, .stack-work/dist/x86_64-osx/Cabal-2.0.0.2/build/Auth/DB/Model/User.o ) ghc: panic! (the 'impossible' happened) (GHC version 8.2.1 for x86_64-apple-darwin): Loading temp shared object failed: dlopen(/var/folders/f8/2_rc4tgd1gj9vbgv7q9gbk4c0000gn/T/ghc45626_0/libghc_261.dylib, 5): no suitable image found. Did find: /var/folders/f8/2_rc4tgd1gj9vbgv7q9gbk4c0000gn/T/ghc45626_0/libghc_261.dylib: malformed mach-o: load commands size (32936) > 32768 Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug }}} Not sure if that's a GHC or stack issue. I can only reproduce it only on a few big projects with 8.2.1 and a bigger number of projects failing with 7.10.3. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:54 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra -------------------------------------+------------------------------------- Reporter: stephenb | Owner: (none) Type: bug | Status: closed Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): dredozubov, since this ticket is getting a bit long could you open another one? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:55 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#12479: build fail of commercialhaskell.com with stack build on mac os x sierra -------------------------------------+------------------------------------- Reporter: stephenb | Owner: (none) Type: bug | Status: closed Priority: highest | Milestone: 8.0.2 Component: Compiler | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: MacOS X | Architecture: Type of failure: Compile-time | Unknown/Multiple crash | Test Case: Blocked By: | Blocking: Related Tickets: #12198 | Differential Rev(s): Phab:D2532 Wiki Page: | -------------------------------------+------------------------------------- Comment (by glguy): There's already a ticket for the problem of the load command size limit added in macOS 10.12 (I don't remember what it was) but it doesn't need a new ticket "malformed mach-o: load commands size (32936) > 32768" -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12479#comment:56 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC