Hi, Am Montag, den 14.07.2014, 11:13 +0000 schrieb Simon Peyton Jones:
I'ts important to monitor the actual bump, rather than the first time the limit is broken. (They aren't the same, of course.)
Quite right. I am in the process of setting up per-commit monitoring of _nofib_ results, and I just added “number of failing tests” to the list of “benchmark numbers”. But that doesn’t help with performance testcases Simple way forward: Always print the results from performance benchmarks (not only when they are out of the limits). Then, when they break the limit, one can go and grep through the old logs for changes. Next step after that: Include the numbers from all performance benchmarks in the same monitoring as the nofib numbers. Technically easy, but I’m not sure how well it scales for the large(?) number of perf tests we have. But one step at a time. Expect an announcement of the performance dashboard thingy soon. Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org