
Hi devs, It seems our CI infrastructure has become too good. My workflow normally involves working with a DEBUG compiler, and then occasionally running the whole testsuite. I don't validate from scratch mid-patch, but it's good to know if I'm failing tests. With HEAD, I'm getting quite a few testsuite failures, and they all seem to be coming from ASSERTs, and they're all problems that I'm pretty sure I didn't cause. My full list of failures (I'm on Mac) is below, but after looking into 5-6 of them and finding only stuff that isn't my fault, I've given up. So: would it be possible to have our CI infrastructure validate DEBUG mode as well? Then it would be easy to spot where these failures are from. backpack/cabal/bkpcabal02/bkpcabal02.run bkpcabal02 [bad stderr] (normal) backpack/cabal/bkpcabal04/bkpcabal04.run bkpcabal04 [bad stderr] (normal) backpack/cabal/bkpcabal03/bkpcabal03.run bkpcabal03 [bad stderr] (normal) backpack/cabal/bkpcabal05/bkpcabal05.run bkpcabal05 [bad stderr] (normal) backpack/cabal/T14304/T14304.run T14304 [bad stderr] (normal) backpack/cabal/bkpcabal06/bkpcabal06.run bkpcabal06 [bad stderr] (normal) backpack/cabal/bkpcabal07/bkpcabal07.run bkpcabal07 [bad stderr] (normal) backpack/cabal/bkpcabal01/bkpcabal01.run bkpcabal01 [bad stderr] (normal) cabal/cabal04/cabal04.run cabal04 [bad stderr] (normal) cabal/T12733/T12733.run T12733 [bad stderr] (normal) cabal/cabal01/cabal01.run cabal01 [bad stderr] (normal) cabal/cabal03/cabal03.run cabal03 [bad stderr] (normal) cabal/cabal09/cabal09.run cabal09 [bad stderr] (normal) cabal/cabal08/cabal08.run cabal08 [bad stderr] (normal) cabal/cabal05/cabal05.run cabal05 [bad stderr] (normal) cabal/cabal06/cabal06.run cabal06 [bad stderr] (normal) dependent/should_compile/T12442.run T12442 [exit code non-0] (normal) dependent/should_compile/T12176.run T12176 [exit code non-0] (normal) driver/T3007/T3007.run T3007 [bad stderr] (normal) gadt/T12087.run T12087 [stderr mismatch] (normal) ghci/scripts/ghci063.run ghci063 [bad stderr] (ghci) indexed-types/should_fail/T13877.run T13877 [stderr mismatch] (normal) parser/should_fail/NumericUnderscoresFail0.run NumericUnderscoresFail0 [stderr mismatch] (normal) parser/should_fail/NumericUnderscoresFail1.run NumericUnderscoresFail1 [stderr mismatch] (normal) patsyn/should_compile/T13350/T13350.run T13350 [bad stderr] (normal) pmcheck/should_compile/T11195.run T11195 [exit code non-0] (normal) polykinds/T14174a.run T14174a [exit code non-0] (normal) printer/T13050p.run T13050p [bad exit code] (normal) safeHaskell/check/pkg01/safePkg01.run safePkg01 [bad stderr] (normal) simplCore/should_compile/T13025.run T13025 [bad stderr] (normal) simplCore/should_compile/T13410.run T13410 [exit code non-0] (normal) typecheck/T13168/T13168.run T13168 [bad stderr] (normal) typecheck/bug1465/bug1465.run bug1465 [bad stderr] (normal) typecheck/should_compile/holes2.run holes2 [exit code non-0] (normal) typecheck/should_compile/valid_substitutions.run valid_substitutions [exit code non-0] (normal) typecheck/should_compile/holes3.run holes3 [stderr mismatch] (normal) typecheck/should_compile/holes.run holes [stderr mismatch] (normal) typecheck/should_compile/T13050.run T13050 [exit code non-0] (normal) typecheck/should_compile/T13822.run T13822 [exit code non-0] (normal) typecheck/should_compile/T14590.run T14590 [exit code non-0] (normal) typecheck/should_compile/T13032.run T13032 [stderr mismatch] (normal) typecheck/should_fail/T7175.run T7175 [stderr mismatch] (normal) Thanks! Richard

Hi, Am Freitag, den 26.01.2018, 22:53 -0500 schrieb Richard Eisenberg:
So: would it be possible to have our CI infrastructure validate DEBUG mode as well? Then it would be easy to spot where these failures are from.
unhelpful comment: Travis used to validate with DEBUG, but it no longer runs the test suite for build time reason. So plausibly we could add a -DEBUG variant to CircleCI? Joachim -- Joachim Breitner mail@joachim-breitner.de http://www.joachim-breitner.de/

Hi, Am Freitag, den 26.01.2018, 23:23 -0500 schrieb Joachim Breitner:
Am Freitag, den 26.01.2018, 22:53 -0500 schrieb Richard Eisenberg:
So: would it be possible to have our CI infrastructure validate DEBUG mode as well? Then it would be easy to spot where these failures are from.
unhelpful comment: Travis used to validate with DEBUG, but it no longer runs the test suite for build time reason. So plausibly we could add a -DEBUG variant to CircleCI?
I now also stumble about the inability to properly test a build built with -DDEBUG. So let me rephrase: Could we please add a -DDEBUG variant to CircleCI? (And then fix the failing tests, or at least mark them as known-to-be- failing on a debugged GHC) Cheers, Joachim -- Joachim Breitner mail@joachim-breitner.de http://www.joachim-breitner.de/

Joachim Breitner
Hi,
Am Freitag, den 26.01.2018, 23:23 -0500 schrieb Joachim Breitner:
Am Freitag, den 26.01.2018, 22:53 -0500 schrieb Richard Eisenberg:
So: would it be possible to have our CI infrastructure validate DEBUG mode as well? Then it would be easy to spot where these failures are from.
unhelpful comment: Travis used to validate with DEBUG, but it no longer runs the test suite for build time reason. So plausibly we could add a -DEBUG variant to CircleCI?
I now also stumble about the inability to properly test a build built with -DDEBUG.
So let me rephrase: Could we please add a -DDEBUG variant to CircleCI?
See D4411.
(And then fix the failing tests, or at least mark them as known-to-be- failing on a debugged GHC)
I'll add this to my list. Cheers, - Ben
participants (3)
-
Ben Gamari
-
Joachim Breitner
-
Richard Eisenberg