On Fri, Apr 11, 2014 at 10:58 AM, Simon Marlow
Let me second this. In particular, I think we regress on GHC performance regularly, because the perf tests just aren't a good way to prevent regressions. When we have a +/- 10% window, someone can commit a 9% regression without triggering a perf failure, but the next patch to come along with a 1% regression will be unfairly blamed.
Agreed. I think graphing results helps here. It's often easier to visualize identify which commit is the real culprit. Aside: I think moving completely to subrepos will generally help us track down regressions, both performance and correctness, faster. Being able to `git bisect` your way to the cause saves a lot of time. Furthermore, by the time we get to the perf tests we're nearly done and
just want to push and go home, not go back and profile GHC. Yet the perf tests have an important purpose: the idea is to catch the problem when we have the crucial piece of information: the patch that caused the regression. Someone can try to optimise GHC later, but they have to start from scratch without the information about what caused the regressions.
Having an automated system to track GHC performance would help a lot with this, I think.
Agree a 100%. Automation is what's needed here.