#12758: Bring sanity to our performance testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: task | Status: new Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): I'm all for eliminating busy-work, but my experience is that we have too few, not too many, performance tests. Mostly, when a perf-test fails, there really is something to investigate, annoying though I often find it. The exception is `max_bytes_used` which wobbles around unpredicatably. I'm not sure how your new approach would make things better. Would you like to elaborate? -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12758#comment:3> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#12758: Bring sanity to our performance testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: Type: task | Status: new Priority: normal | Milestone: 8.2.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"2940a61722b5e5b35421b385dedf07f82c63557d/ghc" 2940a617/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="2940a61722b5e5b35421b385dedf07f82c63557d" testsuite: Specify expected allocations of T12877 for Windows This deviated by 12% from the expected allocations on Windows. Yet another case of #12758. }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12758#comment:5> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#12758: Bring sanity to our performance testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: new Priority: normal | Milestone: 8.4.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * milestone: 8.2.1 => 8.4.1 Comment: There is general agreement that we want to do something along these lines. However, it will probably only happen for 8.4. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12758#comment:7> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#12758: Bring sanity to our performance testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: new Priority: normal | Milestone: 8.6.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): A quick update: Last summer Jared Weakly implemented the proposal of this ticket. The result is Phab:D3758. At the time the plan was to switch to Jenkins in the fall (see #13716) and incorporate his work at that point since it requires a bit of integration with CI. However, shortly before ICFP we started a discussion which culminated in the decision to instead move CI to CircleCI. This process is still on-going (see wiki:ContinuousIntegration) but is nearly an end, at which point we will be able to move ahead with Phab:D3758. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12758#comment:10> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#12758: Bring sanity to our performance testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: new Priority: high | Milestone: 8.6.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"81a5e05d376c075a38e55bc124ea6226c1f3bef7/ghc" 81a5e05/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="81a5e05d376c075a38e55bc124ea6226c1f3bef7" circleci: Skip performance tests Once we finally get the automation for #12758 we can re-enable these. }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12758#comment:12> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#12758: Bring sanity to our performance testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: new Priority: high | Milestone: 8.8.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D3758 Wiki Page: | -------------------------------------+------------------------------------- Comment (by domenkozar): Some insight how I've implemented a similar setting for Snabb (open source high performance networking drivers). Using Nix, we build a matrix of Snabb (multiple git branches, different versions of software dependencies like Qemu, etc) and plot all of those graphs. There is no smart automation to abort of fail for contributions, but if you run these tests before releases - and even as a cronjob every X days, you can spot regressions easily and then bisect the commit. Nix tags what derivations are benchmarks and those run on specialized hardware 1 at the time. Example report: https://hydra.snabb.co/build/3641949/download/2/report.html -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12758#comment:15> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#12758: Bring sanity to our performance testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: new Priority: high | Milestone: 8.8.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D3758, Wiki Page: | Phab:D5059 -------------------------------------+------------------------------------- Changes (by osa1): * differential: Phab:D3758 => Phab:D3758, Phab:D5059 -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12758#comment:17> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#12758: Bring sanity to our performance testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: new Priority: high | Milestone: 8.8.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D3758, Wiki Page: | Phab:D5059 -------------------------------------+------------------------------------- Comment (by Ben Gamari <ben@…>): In [changeset:"932cd41d8c7984c767c1b3b58e05146f69cc5c15/ghc" 932cd41d/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="932cd41d8c7984c767c1b3b58e05146f69cc5c15" testsuite: Save performance metrics in git notes. This patch makes the following improvement: - Automatically records test metrics (per test environment) so that the programmer need not supply nor update expected values in *.T files. - On expected metric changes, the programmer need only indicate the direction of change in the git commit message. - Provides a simple python tool "perf_notes.py" to compare metrics over time. Issues: - Using just the previous commit allows performance to drift with each commit. - Currently we allow drift as we have a preference for minimizing false positives. - Some possible alternatives include: - Use metrics from a fixed commit per test: the last commit that allowed a change in performance (else the oldest metric) - Or use some sort of aggregate since the last commit that allowed a change in performance (else all available metrics) - These alternatives may result in a performance issue (with the test driver) having to heavily search git commits/notes. - Run locally, performance tests will trivially pass unless the tests were run locally on the previous commit. This is often not the case e.g. after pulling recent changes. Previously, *.T files contain statements such as: ``` stats_num_field('peak_megabytes_allocated', (2, 1)) compiler_stats_num_field('bytes allocated', [(wordsize(64), 165890392, 10)]) ``` This required the programmer to give the expected values and a tolerance deviation (percentage). With this patch, the above statements are replaced with: ``` collect_stats('peak_megabytes_allocated', 5) collect_compiler_stats('bytes allocated', 10) ``` So that programmer must only enter which metrics to test and a tolerance deviation. No expected value is required. CircleCI will then run the tests per test environment and record the metrics to a git note for that commit and push them to the git.haskell.org ghc repo. Metrics will be compared to the previous commit. If they are different by the tolerance deviation from the *.T file, then the corresponding test will fail. By adding to the git commit message e.g. ``` # Metric (In|De)crease <metric(s)> <options>: <tests> Metric Increase ['bytes allocated', 'peak_megabytes_allocated'] \ (test_env='linux_x86', way='default'): Test012, Test345 Metric Decrease 'bytes allocated': Test678 Metric Increase: Test711 ``` This will allow the noted changes (letting the test pass). Note that by omitting metrics or options, the change will apply to all possible metrics/options (i.e. in the above, an increase for all metrics in all test environments is allowed for Test711) phabricator will use the message in the description Reviewers: bgamari, hvr Reviewed By: bgamari Subscribers: rwbarton, carter GHC Trac Issues: #12758 Differential Revision: https://phabricator.haskell.org/D5059 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12758#comment:19> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler

#12758: Bring sanity to our performance testsuite -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: task | Status: new Priority: high | Milestone: 8.8.1 Component: Test Suite | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #15936 | Differential Rev(s): Phab:D3758, Wiki Page: | Phab:D5059 -------------------------------------+------------------------------------- Changes (by bgamari): * related: => #15936 -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12758#comment:21> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler