On Mon, 2011-10-03 at 22:59 -0700, Johan Tibell wrote:
Proposal ========
Step 1. Add basic support for building simple benchmark suites e.g. modelled after test suite sections:
Name: foo Version: 1.0 License: BSD3 Cabal-Version: >= 1.9.2 Build-Type: Simple
Benchmark bench-foo type: stdio-1.0 main-is: bench-foo.hs build-depends: base
This doesn't add much on top of normal executable sections, except that it tells Cabal that this isn't an executable it should build by default or install.
Just to point out that the closest workaround at the moment is to add an executable but to mark it as not buildable by default.
Step 2. Add a detailed benchmark interface that allows the benchmark to associate a real valued *measure* (optionally with upper and lower bounds) with a benchmark name. Benchmark naming would work like test naming (e.g. group/subgroup/name). This would allow us to track performance over time.
Step 2 needs to be fleshed out some more, but I think we can get quite a lot of bang-for-the-buck with step 1.
In general, yes I think benchmarking is a sensible follow-on from test suites. We can follow the same approach of specifying a protocol so that we can add more sophisticated interfaces later as we gain experience wit them. Duncan