[GHC] #15758: hsc2hs broken due to incorrect argument passing to the hsc2hs executable

#15758: hsc2hs broken due to incorrect argument passing to the hsc2hs executable -------------------------------------+------------------------------------- Reporter: osa1 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Currently packages with .hsc files can't be built with GHC HEAD on some systems becuase of a problem in hsc2hs wrapper script. I don't understand the problem good enough, but we did some debugging with hvr today and he asked me to file a ticket. Here's how to reproduce: {{{ regex-posix-0.95.2 $ cabal new-build --with- ghc=/home/omer/haskell/ghc/inplace/bin/ghc-stage2 Warning: Unknown/unsupported 'ghc' version detected (Cabal 2.4.0.1 supports 'ghc' version < 8.7): /home/omer/haskell/ghc/inplace/bin/ghc-stage2 is version 8.7.20181017 Warning: The package list for 'hackage.haskell.org' is 22 days old. Run 'cabal update' to get the latest list of available packages. Resolving dependencies... Build profile: -w ghc-8.7.20181017 -O1 In order, the following will be built (use -v for more details): - regex-posix-0.95.2 (lib:regex-posix) (first run) Warning: regex-posix.cabal:6:1: The field "build-type" is specified more than once at positions 6:1, 19:1 Configuring regex-posix-0.95.2... Preprocessing library for regex-posix-0.95.2.. hsc2hs: @/home/omer/haskell/regex-posix-0.95.2/dist- newstyle/build/x86_64-linux/ghc-8.7.20181017/regex- posix-0.95.2/build/Text/Regex/Posix/hsc2hs-response17101-2.txt: openBinaryFile: does not exist (No such file or directory) }}} As far as I understand, the problem is that the script runs `hsc2hs -- @foo` instead of `hsc2hs @foo`. Hopefully hvr will correct if I misunderstood. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15758 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15758: hsc2hs broken due to incorrect argument passing to the hsc2hs executable -------------------------------------+------------------------------------- Reporter: osa1 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by RyanGlScott): * cc: ckoparkar (added) Comment: This looks like it might be related to `hsc2hs`'s recently added support for reading response files. ckoparkar, might you have an idea what's happening here? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15758#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15758: hsc2hs broken due to incorrect argument passing to the hsc2hs executable -------------------------------------+------------------------------------- Reporter: osa1 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by osa1): * cc: hvr (added) Comment: (forgot to cc hvr) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15758#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15758: hsc2hs broken due to incorrect argument passing to the hsc2hs executable -------------------------------------+------------------------------------- Reporter: osa1 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by ckoparkar): Here's another way to reproduce the build failure. If we create a 'Foo.hsc' file, and use {{{hsc2hs}}} built by HEAD to process it via a response file, it fails with the same error. {{{#!haskell -- Foo.hsc main :: IO () main = print $ #size int }}} {{{#!sh $ cat cmd_opts Foo.hsc }}} {{{#!sh $ ~/ghc/inplace/bin/hsc2hs @cmd_opts hsc2hs: @cmd_opts: openBinaryFile: does not exist (No such file or directory) }}} And this would always fail if HEAD is built with GHC < 8.6. I think that it's because in the GHC build system, {{{hsc2hs}}} is built during Phase 0, by the bootstrapping compiler. So if HEAD is bootstrapped with GHC < 8.6, it uses a version of {{{base}}} which does not support response files, and triggers a CPP codepath which does not know how to parse {{{@file}}} command-line arguments. So {{{hsc2hs}}} tries to read a '@file', which does not exist. I am unsure what right solution for this is. We certainly can't move {{{hsc2hs}}} out of 'Phase 0' as some artifacts in 'Phase 1' depend on it. Maybe the final {{{hsc2hs}}} executable that's generated under 'ghc/inplace' could be regenerated in a later phase. What do other people think about this ? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15758#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15758: hsc2hs broken due to incorrect argument passing to the hsc2hs executable -------------------------------------+------------------------------------- Reporter: osa1 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by ckoparkar): I forgot to mention the Cabal bits in comment:3. In Cabal, the decision of whether or not to use response files for invoking {{{hsc2hs}}} is made based only on the version of {{{hsc2hs}}}. {{{#!haskell if hsc2hs_version >= 0.68.4 then use response files else regular command-line arguments }}} But we can have a {{{hsc2hs-0.68.4}}} that is compiled with GHC < 8.6, which won't support response file arguments. I think this is what's happening in the {{{regex-posix}}} example that osa1 posted. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15758#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15758: hsc2hs broken due to incorrect argument passing to the hsc2hs executable -------------------------------------+------------------------------------- Reporter: osa1 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by osa1): OK so if I understand this correctly here's how this happens: - hsc2hs built using bootstrap compiler, which in my case is 8.4.4, which doesn't have `getArgsWithResponseFiles` function. - So in [https://github.com/haskell/hsc2hs/commit/0535fe65467d1a07b838c17c62438086c12... this commit] the code with `getArgs` is used. - However, Cabal doesn't consider this possibility that a new enough GHC is built using a GHC without `getArgsWithResponseFiles`, and just based on hsc2hs version thinks that it should support `getArgsWithResponseFiles`, and passes arguments usign response file conventions. The question is: is this a GHC bug or a Cabal bug? - If we want newer GHCs (HEAD, 8.8) to always support response files this is a GHC bug. - Otherwise this is a Cabal bug.
We certainly can't move hsc2hs out of 'Phase 0' as some artifacts in 'Phase 1' depend on it
What do you mean by "Phase"? I think we can build hsc2hs using stage 1 compiler and libs and that'd fix the problem, no? Any .hsc files for stage 1 can be compiled using bootstrap hsc2hs. Here's the hack I used to fix this on Cabal side: {{{ diff --git a/Cabal/Distribution/Simple/PreProcess.hs b/Cabal/Distribution/Simple/PreProcess.hs index daf419ef2..da231f9a0 100644 --- a/Cabal/Distribution/Simple/PreProcess.hs +++ b/Cabal/Distribution/Simple/PreProcess.hs @@ -396,7 +396,7 @@ ppHsc2hs bi lbi clbi = (hsc2hsProg, hsc2hsVersion, _) <- requireProgramVersion verbosity hsc2hsProgram anyVersion (withPrograms lbi) -- See Trac #13896 and https://github.com/haskell/cabal/issues/3122. - let hsc2hsSupportsResponseFiles = hsc2hsVersion >= mkVersion [0,68,4] + let hsc2hsSupportsResponseFiles = False pureArgs = genPureArgs gccProg inFile outFile if hsc2hsSupportsResponseFiles then withResponseFile }}} I'd suggest bumping priority, but I guess we should first decide whether to fix this on GHC side or Cabal side. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15758#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15758: hsc2hs broken due to incorrect argument passing to the hsc2hs executable -------------------------------------+------------------------------------- Reporter: osa1 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by hvr): It's remarkable how many bugs this issue entails... :-) I've quickly addressed part of the issue via https://github.com/haskell/hsc2hs/commit/8807b4cd9b9efc719828b52cd9aecb9892d... (one problem to consider is that we have at least two lib:Cabal releases out there eligible to custom Setup.hs scripts as well as included in the cabal-install-2.4.0.0 release which have the `>= 0.68.4` logic hardwired; so the metadata revision is the most economical way to mitigate that issue) So the next thing that needs fixing is the hsc2hs wrapper script used for inplace; moreoever I noticed yet another bug that wasn't mentioned here: GHC also installs a wrapper script into its installed `bin/` folder, e.g. `/opt/ghc/8.6.1/bin/hsc2hs`: {{{#!bash #!/bin/sh exedir="/opt/ghc/8.6.1/lib/ghc-8.6.1/bin" exeprog="hsc2hs" executablename="$exedir/$exeprog" datadir="/opt/ghc/8.6.1/share" bindir="/opt/ghc/8.6.1/bin" topdir="/opt/ghc/8.6.1/lib/ghc-8.6.1" HSC2HS_EXTRA="--cflag=-fno-stack-protector --lflag=-fuse-ld=gold" #!/bin/sh tflag="--template=$topdir/template-hsc.h" Iflag="-I$topdir/include/" for arg do case "$arg" in # On OS X, we need to specify -m32 or -m64 in order to get gcc to # build binaries for the right target. We do that by putting it in # HSC2HS_EXTRA. When cabal runs hsc2hs, it passes a flag saying which # gcc to use, so if we set HSC2HS_EXTRA= then we don't get binaries # for the right platform. So for now we just don't set HSC2HS_EXTRA= # but we probably want to revisit how this works in the future. # -c*) HSC2HS_EXTRA=;; # --cc=*) HSC2HS_EXTRA=;; -t*) tflag=;; --template=*) tflag=;; --) break;; esac done exec "$executablename" ${tflag:+"$tflag"} $HSC2HS_EXTRA ${1+"$@"} "$Iflag" }}} while this script doesn't inject any `--`s before response-file args, it exhibits a different problem: it's logic is completely bypassed when you use response files, as the script then cannot rewrite the flags because it doesn't look into the response files! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15758#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15758: hsc2hs broken due to incorrect argument passing to the hsc2hs executable -------------------------------------+------------------------------------- Reporter: osa1 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D5238 Wiki Page: | -------------------------------------+------------------------------------- Changes (by ckoparkar): * differential: => Phab:D5238 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15758#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15758: hsc2hs broken due to incorrect argument passing to the hsc2hs executable -------------------------------------+------------------------------------- Reporter: osa1 | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D5238 Wiki Page: | -------------------------------------+------------------------------------- Comment (by klebinger.andreas@…>): In [changeset:"fce07c99fa6528e95892604edb73fb975d6a01fc/ghc" fce07c99/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="fce07c99fa6528e95892604edb73fb975d6a01fc" Update hsc2hs submodule to work around bug in response file handling. Summary: This works around #15758 by bumping hsc2hs. The new version includes support for response files independent of the boot compiler. Test Plan: ci, building formerly broken packages Reviewers: bgamari, RyanGlScott, ckoparkar Reviewed By: ckoparkar Subscribers: rwbarton, carter GHC Trac Issues: #15758 Differential Revision: https://phabricator.haskell.org/D5250 }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15758#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#15758: hsc2hs broken due to incorrect argument passing to the hsc2hs executable -------------------------------------+------------------------------------- Reporter: osa1 | Owner: (none) Type: bug | Status: closed Priority: normal | Milestone: Component: Compiler | Version: Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D5238, Wiki Page: | Phab:D5250 -------------------------------------+------------------------------------- Changes (by ckoparkar): * status: new => closed * differential: Phab:D5238 => Phab:D5238, Phab:D5250 * resolution: => fixed Comment: Now that Phab:D5238 and Phab:D5250 have landed, and hsc2hs-0.68.4 has been updated to have the correct revision on Hackage, I believe we've addressed all the issues uncovered by this ticket. I verified that I can build {{{regex-posix}}} with GHC HEAD, and also the simple {{{Foo.hsc}}} example from comment:3 works. I'll go ahead and close this. (Let's re-open if there are more things that need to be done.) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15758#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC