[GHC] #11165: Testsuite framework failures are too easy to ignore and too hard to find
#11165: Testsuite framework failures are too easy to ignore and too hard to find -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 7.10.2 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- The testsuite driver really should do a better job of drawing the user's attention to framework failures. The only evidence in the testsuite summary that something went wrong is a non-zero "framework failures" count. If you notice that this count is non-zero then you need to sift manually through the initial log output produced by the driver and find the culprit. It's just generally terrible. Testsuite failures are failures, we should at least treat them as such and include them in the `TEST=...` message in the testsuite summary so they can be easily reproduced. Ideally we would also save the exception that caused the failure and show it in the summary as well. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11165 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#11165: Testsuite framework failures are too easy to ignore and too hard to find -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: patch Priority: normal | Milestone: Component: Test Suite | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2026 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => patch * differential: => Phab:D2026 Comment: Phab:D2026 teaches the testsuite driver to at least show the names of the failing tests in the summary. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11165#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#11165: Testsuite framework failures are too easy to ignore and too hard to find -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: patch Priority: normal | Milestone: Component: Test Suite | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2026 Wiki Page: | -------------------------------------+------------------------------------- Comment (by goldfire): Hooray! I just got caught by this trap. Thanks. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11165#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#11165: Testsuite framework failures are too easy to ignore and too hard to find
-------------------------------------+-------------------------------------
Reporter: bgamari | Owner:
Type: bug | Status: patch
Priority: normal | Milestone:
Component: Test Suite | Version: 7.10.2
Resolution: | Keywords:
Operating System: Unknown/Multiple | Architecture:
| Unknown/Multiple
Type of failure: None/Unknown | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s): Phab:D2026
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Ben Gamari
#11165: Testsuite framework failures are too easy to ignore and too hard to find
-------------------------------------+-------------------------------------
Reporter: bgamari | Owner:
Type: bug | Status: patch
Priority: normal | Milestone:
Component: Test Suite | Version: 7.10.2
Resolution: | Keywords:
Operating System: Unknown/Multiple | Architecture:
| Unknown/Multiple
Type of failure: None/Unknown | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s): Phab:D2026
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Thomas Miedema
#11165: Testsuite framework failures are too easy to ignore and too hard to find -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 7.10.2 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2026 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: patch => closed * resolution: => fixed * milestone: => 8.2.1 Comment: I think it is safe to call this fixed. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11165#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC