#12758: Bring sanity to our performance testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: task | Status: new Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Description changed by bgamari: @@ -13,1 +13,5 @@ - * + * We replace the existing `compiler_stats_num_field` test modifier with + an implementation that dumps performance metrics to a CSV file + * Introduce a tool to compare this CSV file to metrics associated with a + ancestor commit via `git notes` + * The tool would also be able to add notes New description: The GHC testsuite's performance tests tends to be a constant source of busy-work as it requires that contributors manually bump performance numbers and propagate these changes across platforms. Moreover, the testsuite is poor at noticing performance regressions due to false positive failures (due to spurious environmental differences) and false negatives (due to the rather generous acceptance windows that many tests have). Joachim and I briefly discussed this at Hac Phi and came up with the following proposal: * We rip expected performance numbers out of the `.T` files * We replace the existing `compiler_stats_num_field` test modifier with an implementation that dumps performance metrics to a CSV file * Introduce a tool to compare this CSV file to metrics associated with a ancestor commit via `git notes` * The tool would also be able to add notes -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12758#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler