[GHC] #11165: Testsuite framework failures are too easy to ignore and too hard to find
#11165: Testsuite framework failures are too easy to ignore and too hard to find -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Test Suite | Version: 7.10.2 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- The testsuite driver really should do a better job of drawing the user's attention to framework failures. The only evidence in the testsuite summary that something went wrong is a non-zero "framework failures" count. If you notice that this count is non-zero then you need to sift manually through the initial log output produced by the driver and find the culprit. It's just generally terrible. Testsuite failures are failures, we should at least treat them as such and include them in the `TEST=...` message in the testsuite summary so they can be easily reproduced. Ideally we would also save the exception that caused the failure and show it in the summary as well. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11165> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11165: Testsuite framework failures are too easy to ignore and too hard to find -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: patch Priority: normal | Milestone: Component: Test Suite | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2026 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => patch * differential: => Phab:D2026 Comment: Phab:D2026 teaches the testsuite driver to at least show the names of the failing tests in the summary. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11165#comment:1> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11165: Testsuite framework failures are too easy to ignore and too hard to find -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: patch Priority: normal | Milestone: Component: Test Suite | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2026 Wiki Page: | -------------------------------------+------------------------------------- Comment (by goldfire): Hooray! I just got caught by this trap. Thanks. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11165#comment:2> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11165: Testsuite framework failures are too easy to ignore and too hard to find -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: patch Priority: normal | Milestone: Component: Test Suite | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2026 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"d0787a2c232b61f8be080048ea48af7697094c97/ghc" d0787a2/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="d0787a2c232b61f8be080048ea48af7697094c97" testsuite: Identify framework failures in testsuite summary Currently the testsuite driver tells you how many tests failed due to a framework failure but you need to manually grep through the testsuite output to identify which ones. Test Plan: Validate with, e.g., a timing out testcase Reviewers: austin, thomie Reviewed By: austin, thomie Differential Revision: https://phabricator.haskell.org/D2026 GHC Trac Issues: #11165 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11165#comment:3> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11165: Testsuite framework failures are too easy to ignore and too hard to find -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: patch Priority: normal | Milestone: Component: Test Suite | Version: 7.10.2 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2026 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Thomas Miedema <thomasmiedema@…>): In [changeset:"782cacf57300908d8a608bac7d26be59586f2af0/ghc" 782cacf/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="782cacf57300908d8a608bac7d26be59586f2af0" Testsuite: framework failure improvements (#11165) * add framework failures to unexpected results list * report errors in .T files as framework failures (show in summary) * don't report missing tests when framework failures in .T files }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11165#comment:4> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11165: Testsuite framework failures are too easy to ignore and too hard to find -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: bug | Status: closed Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 7.10.2 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D2026 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: patch => closed * resolution: => fixed * milestone: => 8.2.1 Comment: I think it is safe to call this fixed. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11165#comment:5> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
participants (1)
-
GHC