GHC silently turns off dynamic output, should this be an error?
Hello, I’ve investigated an issue (#9176) where ghc with --make -dynamic-too appears to run successfully but doesn’t generate some .dyn_hi files. The cause seems to be that GHC silently turns off dynamic-too output when it finds an inconsistency between the .hi and .dyn_hi interface hashes of an imported module. The symptom, for me, is that some cabal packages will build but fail to install. The real problem is most likely not in the failing package, but in an imported module from a package that’s already installed. I’m inclined to say that GHC should stop with an error in this situation, which would produce a visible error when the file fails to compile. However, since GHC carefully avoids bailing out, I wonder if I’m missing something. Is there a good reason to turn off dynamic-too output instead of reporting an error? Additionally, is it ever valid to have a pair of .hi and .dyn_hi files with different interface hashes? Or is it invalid data, in the same way that a corrupted .hi file is invalid data? I’m inclined to say that it is invalid data, because -dynamic-too promises to generate two different (static and dynamic) implementations of the same Haskell code, taking two different implementations of the same imported Haskell modules as input. If the inputs aren’t really two different implementations of the same Haskell code, you’ve somehow provided invalid input.
I wonder if this is the source of the problem I had with ^C'ing parallel rebuilds. I'll often build, realizing I should be doing something differently, ^C out, tweak something start a build again, etc. but if I have -j on then I find that doing so occasionally causes me to start missing dyn_hi files after the package builds. If I ^C'd in between generating one and the other being written, could that be what is causing other consistency problems later in my build? I've hesitated on reporting it (other than mentioning it in passing to Austin) because it has been fiendishly hard to reproduce and only happens on my more er.. monolithic projects. I've gone back to sequential builds as a result of this though. -Edward On Mon, Jun 23, 2014 at 1:58 PM, Christopher Rodrigues < pointlessmonad@gmail.com> wrote:
Hello,
I’ve investigated an issue (#9176) where ghc with --make -dynamic-too appears to run successfully but doesn’t generate some .dyn_hi files. The cause seems to be that GHC silently turns off dynamic-too output when it finds an inconsistency between the .hi and .dyn_hi interface hashes of an imported module. The symptom, for me, is that some cabal packages will build but fail to install. The real problem is most likely not in the failing package, but in an imported module from a package that’s already installed.
I’m inclined to say that GHC should stop with an error in this situation, which would produce a visible error when the file fails to compile. However, since GHC carefully avoids bailing out, I wonder if I’m missing something. Is there a good reason to turn off dynamic-too output instead of reporting an error?
Additionally, is it ever valid to have a pair of .hi and .dyn_hi files with different interface hashes? Or is it invalid data, in the same way that a corrupted .hi file is invalid data? I’m inclined to say that it is invalid data, because -dynamic-too promises to generate two different (static and dynamic) implementations of the same Haskell code, taking two different implementations of the same imported Haskell modules as input. If the inputs aren’t really two different implementations of the same Haskell code, you’ve somehow provided invalid input.
_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
On Mon, Jun 23, 2014 at 12:58:16PM -0500, Christopher Rodrigues wrote:
Additionally, is it ever valid to have a pair of .hi and .dyn_hi files with different interface hashes?
You can, for example, compile package foo the vanilla way with -O, and the dynamic way without -O. You'll then get mismatched hashes. If you then try to compile bar (which depends on foo) with -dynamic-too then the idea was that it would transparently fall back to compiling the two ways separately. Otherwise every build system (Cabal, make rules, etc) that wants to use -dynamic-too would have to handle this failure and either fail or fall back itself. Thanks Ian
On Jun 24, 2014 1:36 PM, "Ian Lynagh"
On Mon, Jun 23, 2014 at 12:58:16PM -0500, Christopher Rodrigues wrote:
Additionally, is it ever valid to have a pair of .hi and .dyn_hi files
with
different interface hashes?
You can, for example, compile package foo the vanilla way with -O, and the dynamic way without -O. You'll then get mismatched hashes.
Also, any file that uses Template Haskell could potentially generate different code at each compilation. But IMHO this isn't something ghc should try to support. John L.
Investigating #9176 further, GHC does at least three different things with imported modules under -dynamic-too compilation. It probably should be changed to handle imports uniformly. It looks like dynamic-too support was bolted on to a system designed for one-pass compilation, and I'm not clear on how to restructure it. Guidance would be appreciated, as it would cut down the time I need to figure out how the driver works. Here are the three things that can happen: 1. When compiling in one-shot mode, with -c, GHC will do what Igloo described. If normal and dynamic interface hashes don't match, it will compile the normal way, then compile again the dynamic way. 2. When compiling with --make and importing from a package, if normal and dynamic interface hashes don't match, GHC will disable dynamic-too, but it will not restart the pipeline. Only the normal .hi file is produced. 3. When compiling with --make and importing from a non-package module, GHC will not check interface hashes. It will proceed as if normal and dynamic interface hashes match. -Chris
On 06/07/2014 12:25, Christopher Rodrigues wrote:
Investigating #9176 further, GHC does at least three different things with imported modules under -dynamic-too compilation. It probably should be changed to handle imports uniformly. It looks like dynamic-too support was bolted on to a system designed for one-pass compilation, and I'm not clear on how to restructure it. Guidance would be appreciated, as it would cut down the time I need to figure out how the driver works.
Here are the three things that can happen:
1. When compiling in one-shot mode, with -c, GHC will do what Igloo described. If normal and dynamic interface hashes don't match, it will compile the normal way, then compile again the dynamic way.
2. When compiling with --make and importing from a package, if normal and dynamic interface hashes don't match, GHC will disable dynamic-too, but it will not restart the pipeline. Only the normal .hi file is produced.
This does sound wrong.
3. When compiling with --make and importing from a non-package module, GHC will not check interface hashes. It will proceed as if normal and dynamic interface hashes match.
This might be OK if we have already compiled all the modules in this compilation pass. If some were compiled previously, then it is possible that we have some non-matching .hi files, and in that case I think we should probably just consider the module out of date and recompile it, generating both dynamic and non-dynamic .hi files, which will be guaranteed to have the same hash. Cheers, Simon
participants (5)
-
Christopher Rodrigues -
Edward Kmett -
Ian Lynagh -
John Lato -
Simon Marlow