There are a few "projects" in this area I think we should undertake: ## Set up a performance dashboard Example: http://goperfd.appspot.com/ (try clicking the "Perf" and "Graphs" buttons.) I think the way to go here is: * Figure out how to build GHC and run the nofib suite in a repeatable (i.e. scripted) manner. * Set up a Jenkins build bot on a "quiet" machine. Have it run the above script in "exclusive" mode (i.e. no other Jenkins jobs can run in parallel). * Write a script that gathers the nofib output and sticks it in a database. The database should be keyed by the commit hash. Use a mature DB (e.g. MySQL, sqlite, postgress). * Write a little web frontend that graphs the results over time. Don't reinvent the wheel/shave all the Yaks. You might be able to reuse the Go frontend and run it on appengine. * Bonus points: have the jenkins job send an email if performance regresses above some threshold. There might already exist Jenkins plugins that can help with the last 4 steps. ## Improve our benchmarks Many of the benchmarks in nofib have too short runtime nowadays to be accurate enough when comparing the performance of two GHC builds. The shootout benchmarks are good (I've checked), but the remaining ones should be considered suspicious. We ought to weed out or improve benchmarks that are no longer accurate. This might involve having the run on bigger inputs and thus run for longer. In addition, running the existing benchmark suites of some core libraries (e.g. from the HP) would also be very useful. The difficulty is to get Criterion to build with HEAD reliably. ## One-off nofib run against the lastest N major GHC releases To see if we have already regressed (and see where we have regressed) I think we should just run the current nofib suite (probably using the "slow" mode) against 6.12 and up to see where we have already regressed. File a bug for each regression, optionally with a small analysis of why we regressed (e.g. look at the Core and the output of +RTS -s). Cheers, Johan