[GHC] #16242: Hadrian is too aggressive in rebuilding
#16242: Hadrian is too aggressive in rebuilding -------------------------------------+------------------------------------- Reporter: goldfire | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.6.3 (Hadrian) | Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Here's what has happened: 1. `build` (with settings `devel2`). This did its thing successfully. 2. Edit a comment in one file. 3. `build test --only=blahblahblah`. I was expecting to run a test. (It was a new test-case, a change unrelated to the comment I fixed.) What happens though is that stage-1 is rebuilt, and now I'm currently waiting for the entire RTS to be rebuilt. And all I wanted to do is run a test! I understand why Hadrian is doing this; it's technically got the upper hand here. But it's very frustrating. I suppose I could have done `--freeze1`, but it's too late now. And, even if I didn't do `--freeze1`, I'm surprised that the whole RTS needs to be rebuilt. Again, I know why Hadrian wants to. But I miss the old `LAX_DEPENDENCIES` feature of `make`. So, I suppose this boils down to two requests: 1. Restore the `LAX_DEPENDENCIES` feature. I don't know the specification of that feature, but my experience is that it just did the Right Thing, and I always had it enabled (except during validation). 2. Have some way of just running a test, without any rebuilding. Somewhat separately, it might be nice to package up common idioms into smaller command lines. Again, individual devs can do this, but centralized support (especially so that these scripts can be run from a variety of directories) would be nice. For example, I imagine my suggestion (2) above will lead to `build test --no-recompilation --only=xyz`, but that's a long line to write! Thanks! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16242 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#16242: Hadrian is too aggressive in rebuilding -------------------------------------+------------------------------------- Reporter: goldfire | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.6.3 (Hadrian) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by snowleopard):
Edit a comment in one file. [...] What happens though is that stage-1 is rebuilt
Restore the LAX_DEPENDENCIES feature. I don't know the specification of
Wait, this doesn't sound right. Which file did you edit? Here is what should have happened: the corresponding compiler should have been rerun generating the same object file, at which point the rebuild should have stopped (we call this feature "early cutoff"). Are we hitting GHC non- determinism here, i.e. the resulting object file was different after editing a comment? that feature, but my experience is that it just did the Right Thing Although I agree that having a flag that does the Right Thing would be useful, I'd like to point out that Hadrian provides a general mechanism for skipping the rebuild of any collection of files. `--freeze1` simply uses this mechanism by skipping everything in `stage0//*`. Here is the corresponding code fragment: https://gitlab.haskell.org/ghc/ghc/blob/10faf44d97095b2f8516b6d449d266f6889d... You can specify the files that should be ignored from the command line using `--skip` flag. For example, the equivalent of the flag `--freeze1` is `--skip=stage0//*`.
Somewhat separately, it might be nice to package up common idioms into smaller command lines.
I think "build flavours" are the intended mechanism for common idioms, although typing `--flavour=...` is indeed inconvenient. We could add a shorthand `-f` for this. Separate scripts for such idioms sounds like a plausible approach too, although there is a risk that having multiple ways to do the same thing may be confusing. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16242#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#16242: Hadrian is too aggressive in rebuilding -------------------------------------+------------------------------------- Reporter: goldfire | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.6.3 (Hadrian) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by goldfire): I edited `Coercion.hs`. But, even if I had edited it in a meaningful way (instead of just editing a comment), I'm surprised to see the whole RTS rebuilt. I believe that's what `LAX_DEPENDENCIES` does for `make`: it assumes that, just because stage-1 has changed, it doesn't mean we need to rebuild everything that stage-1 built. Moreover, I really just wanted to run a test. So, I suppose I could do that with `build --skip=stage0//* --skip=stage1//* test --only=xyz`. Again, that's quite a mouthful. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16242#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#16242: Hadrian is too aggressive in rebuilding -------------------------------------+------------------------------------- Reporter: goldfire | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.6.3 (Hadrian) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by snowleopard): * cc: snowleopard, alpmestan (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16242#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#16242: Hadrian is too aggressive in rebuilding -------------------------------------+------------------------------------- Reporter: goldfire | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.6.3 (Hadrian) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): IIRC part of the reason that the `make` build system behaves "correctly" here is that the testsuite rule is essentially decoupled from the rest of the build system: it has no dependencies whatsoever. You can observe this by running `make test` in a clean tree. You will find that it makes no attempt to build anything but rather just starts in on running the testsuite, using whatever GHC it finds on the path. There is actually a good reason for this: the testsuite can be run against any GHC, not only one from a GHC working directory. As to whether Hadrian should emulate this, I'm torn. Certainly we should have **some** target for running the testsuite against an arbitrary out- of-tree GHC. However, it would make sense if `hadrian test` tested the current working directory. I think the real root of the problem here is that it's very easy to forget adding `--freeze1` to the command line. Perhaps having a way to persist this setting would alleviate much of the pain associated with unintended rebuilds. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16242#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
I think the real root of the problem here is that it's very easy to forget adding `--freeze1` to the command line. Perhaps having a way to
#16242: Hadrian is too aggressive in rebuilding -------------------------------------+------------------------------------- Reporter: goldfire | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.6.3 (Hadrian) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by snowleopard): persist this setting would alleviate much of the pain associated with unintended rebuilds. We could indeed make it possible to set `--freeze1`, or more generally `--skip` any set of dependencies, by modifying the default build flavour in `UserSettings.hs`. This requires the user to make some initial changes in their `UserSettings` file, but after that any `build` command will rebuild exactly what they asked. Does this sound like the right approach? I think implementation-wise, this is quite easy. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16242#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#16242: Hadrian is too aggressive in rebuilding -------------------------------------+------------------------------------- Reporter: goldfire | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.6.3 (Hadrian) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by goldfire): Can we also make a `--freeze2` setting, then? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16242#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#16242: Hadrian is too aggressive in rebuilding -------------------------------------+------------------------------------- Reporter: goldfire | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.6.3 (Hadrian) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by snowleopard):
Can we also make a `--freeze2` setting, then?
Do you mean add a command line flag `--freeze2`, or allow the user to achieve the same effect in `UserSettings`? Both are relatively easy, but I'd like to clarify what you meant exactly. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16242#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#16242: Hadrian is too aggressive in rebuilding -------------------------------------+------------------------------------- Reporter: goldfire | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.6.3 (Hadrian) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by goldfire): Both. Once all these features land, I anticipate spending some time creating my own scripts to make it easy to, e.g., switch flavors and specify these flags. Then, I should be able to customize the workflow nicely. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16242#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
Does this sound like the right approach? I think implementation-wise,
#16242: Hadrian is too aggressive in rebuilding -------------------------------------+------------------------------------- Reporter: goldfire | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.6.3 (Hadrian) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): this is quite easy. Sounds like a good start to me. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16242#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
As to whether Hadrian should emulate this, I'm torn. Certainly we should have some target for running the testsuite against an arbitrary out-of-
#16242: Hadrian is too aggressive in rebuilding -------------------------------------+------------------------------------- Reporter: goldfire | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Build System | Version: 8.6.3 (Hadrian) | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by alpmestan): tree GHC. However, it would make sense if hadrian test tested the current working directory. FWIW, I have implemented this as part of my work to tweak the `validate` script to support hadrian. Required several changes. I'll push a MR tomorrow I think. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/16242#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC