[GHC] #15022: Stackage HEAD detected a suspicious change
#15022: Stackage HEAD detected a suspicious change -------------------------------------+------------------------------------- Reporter: mrkkrp | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Here it goes: {{{ ==== SUSPICIOUS CHANGES stm-delay: at nightly-2018-04-05-5161609117c16cb7b29b2b8b1cd41e74341d4137.csv build succeeded, 2 test suites passed, 0 test suites failed at nightly-2018-04-05-3cfb12d8adac37e5565d66fd173e4648cc041e65.csv build succeeded, 1 test suites passed, 0 test suites failed There are changes that need attention of GHC team. }}} This not necessarily indicates a bug or regression, but the fact is, between these two commits 1 test suite stopped building. Maybe it's worth some investigation. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15022 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15022: Stackage HEAD detected a suspicious change -------------------------------------+------------------------------------- Reporter: mrkkrp | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mrkkrp): Version of stm-delay is 0.1.1.1, the version is determined by the build plan, nigthly-2018-04-05, found here: https://github.com/fpco/stackage-nightly -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15022#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15022: Stackage HEAD detected a suspicious change -------------------------------------+------------------------------------- Reporter: mrkkrp | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by hvr): So, the difference is merely
2 test suites passed, 0 test suites failed
vs
1 test suites passed, 0 test suites failed
since for both cases, no test suites failed; does this mean that the test suite failed to build to due a compile failure, or that it merely wasn't even attempted to be built because no install-plan could be found? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15022#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15022: Stackage HEAD detected a suspicious change -------------------------------------+------------------------------------- Reporter: mrkkrp | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by RyanGlScott): Indeed, this ticket is incredibly vague. What exactly is going on here? Is this using GHC HEAD? GHC 8.4.1? Something else? The use of the phrase "nightly" does not tell me much. Also, you give commit hashes, which I can't tell are from GHC or `stm-delay`. Are these build logs hosted online somewhere? Is there a list of which packages are supposed to build and supposed to fail? (If this really //is// GHC HEAD, it would seem likely that you'd run into spurious failures more often.) How is it determined that this build failure is GHC's fault and not the fault of say, Stackage or a downstream dependency. In short, I'm totally lost. Some context here would be desperately appreciated. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15022#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15022: Stackage HEAD detected a suspicious change -------------------------------------+------------------------------------- Reporter: mrkkrp | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mpickering): Mark made a post to the devops list but not the main ghc-devs list which explains some more context. https://mail.haskell.org/pipermail/ghc-devops-group/2018-April/000182.html -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15022#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15022: Stackage HEAD detected a suspicious change -------------------------------------+------------------------------------- Reporter: mrkkrp | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mrkkrp): Sorry, for throwing it here without sufficient explanations. Here are some clarifications: https://mail.haskell.org/pipermail/ghc-devops-group/2018-April/000182.html And a blog post that describes everything in details is coming. If the sum of succeeding/failing test suites is decreases this indicates that some test suites failed to build. Hashes are GHC commit hashes on master branch. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15022#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15022: Stackage HEAD detected a suspicious change -------------------------------------+------------------------------------- Reporter: mrkkrp | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mrkkrp):
How is it determined that this build failure is GHC's fault and not the fault of say, Stackage or a downstream dependency?
The build plan we use is fixed, so versions of packages and their dependencies cannot change. Any change observed is thus a result of some change in GHC. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15022#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15022: Stackage HEAD detected a suspicious change -------------------------------------+------------------------------------- Reporter: mrkkrp | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by hvr): One thing's that's utterly lost on me: How am I supposed to make any use of this by merely using cabal? ;-) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15022#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15022: Stackage HEAD detected a suspicious change -------------------------------------+------------------------------------- Reporter: mrkkrp | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by RyanGlScott): Perhaps another way of stating hvr's question is: are there logs available which show how these packages were built, and moreover, show what the build failure was? It would certainly save GHC developers some faffing about if there was a clear indication of //why// the package failed to build. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15022#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15022: Stackage HEAD detected a suspicious change -------------------------------------+------------------------------------- Reporter: mrkkrp | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by nomeata): I guess the process is not fully smoothed out yet, but that is expected. Nevertheless I want to say: **This is great! ** This is going to make GHC development much more robust and better!
The build takes about one hour.
Wow, that’s much less than I thought. that’s faster than `perf.haskell.org`… Can we run it on every commit then? Or, maybe a bit less wasteful: Run it every few hours, but if there is a difference, then automatically schedule additional runs of the commits in between to bisect the change? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15022#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15022: Stackage HEAD detected a suspicious change -------------------------------------+------------------------------------- Reporter: mrkkrp | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by RyanGlScott): Case in point for why build logs are critical here: I just tried running the test suites for `stm-delay-0.1.1.1` using commit 3cfb12d8adac37e5565d66fd173e4648cc041e65 using a standard: {{{ $ cabal configure -w ~/Software/ghc/inplace/bin/ghc-stage2 --enable-tests $ cabal test }}} And both test suites successfully built and ran for me. So clearly `stackage-head` is doing something different, but I have no idea what that could be. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15022#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15022: Stackage HEAD detected a suspicious change -------------------------------------+------------------------------------- Reporter: mrkkrp | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mrkkrp): Good points about the logs. Right now we use information that stackage- curator gives us which isn't plenty. This probably can be improved. I'll investigate the std-delay failure tomorrow and if necessary download the actual logs from the cache. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15022#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15022: Stackage HEAD detected a suspicious change -------------------------------------+------------------------------------- Reporter: mrkkrp | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Regardless of what happens with this particular ticket, I would like to reiterate nomeata's message: this sort of tooling is long overdue. Thank you for picking this up, mrkkrp! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15022#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15022: Stackage HEAD detected a suspicious change -------------------------------------+------------------------------------- Reporter: mrkkrp | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mboes): Replying to [comment:9 nomeata]:
The build takes about one hour.
Wow, that’s much less than I thought.
that’s faster than `perf.haskell.org`… Can we run it on every commit
It's only building a bit more than half of all Stackage Nightly right now, because 5 packages currently fail and those are dependencies for the rest. We expect that all told, on the "xlarge" CircleCI machine types, a full Stackage build should take around 2 hours. That's just an extrapolation though. Reality might be quite a bit worse (or better). then?
Or, maybe a bit less wasteful: Run it every few hours, but if there is a
difference, then automatically schedule additional runs of the commits in between to bisect the change? It's currently set at running every 6 hours, and only using GHC master. But I agree, running it at every commit could well be viable. Let's iterate on this as we get more experience using it. :) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15022#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#15022: Stackage HEAD detected a suspicious change -------------------------------------+------------------------------------- Reporter: mrkkrp | Owner: (none) Type: bug | Status: closed Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.2.2 Resolution: invalid | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by mrkkrp): * status: new => closed * resolution: => invalid Comment: It looks like this was a false alarm. I re-run the script with the older GHC bindist and got single test suite run. Something has changed but the change is not caused by GHC. So, one test suite (test-threaded) is not run anymore for unknown reason. The other test suite (test) is run. They are almost identical BTW: {{{ test-suite test type: exitcode-stdio-1.0 hs-source-dirs: test main-is: Main.hs ghc-options: -Wall -fno-warn-missing-signatures -fno-warn-name-shadowing -fno-warn-unused-do-bind -fno-warn-unused-matches build-depends: base >= 4.3 && < 5 , stm , stm-delay test-suite test-threaded type: exitcode-stdio-1.0 hs-source-dirs: test main-is: Main.hs ghc-options: -Wall -threaded -fno-warn-missing-signatures -fno-warn-name-shadowing -fno-warn-unused-do-bind -fno-warn-unused-matches build-depends: base >= 4.3 && < 5 , stm , stm-delay }}} There was also a bug in detection of failing packages (those that failing not during building but during configuration step were not included in the report at all), there 27 failing packages, but this is not relevant in this case. I have adjusted the setup to exclude any changes but those introduced by GHC: * We previously downloaded `stack` in binary form on every run without controlling whether it's the same `stack` every time. `stack` binary is now fixed and cannot change between runs. * The same for `stackage-curator`. * The docker image we used `snoyberg/stackage:nightly` also changes frequently and so I created a different image that I won't update unless needed. * We previously run `stack update` on every execution of the script, which may have affected the results. I included this command in `Dockerfile` instead so updates of Hackage indices are controlled as well. This should eliminate influence of external factors on results that we observe. So, again, sorry for the false alarm! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15022#comment:14 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC