Hi all, Problem ======= After having used Makefiles to build my benchmark suites for quite some time I'm convinced we need support for building benchmark using Cabal. Make doesn't solve the problem of building a benchmarks particularly well. There are several problems: * Building your library using Cabal and your benchmarks using Make sometimes ends up with different package versions being used for the two separate builds, resulting in compilation errors. * Lots of functionality that's already in Cabal is duplicated in Makefiles (e.g. preprocessor rules). * It's hard to get proper incremental compilation; I always use `make -C benchmarks clean all` to rebuild the benchmarks, just to make sure that everything is up-to-date. Proposal ======== Step 1. Add basic support for building simple benchmark suites e.g. modelled after test suite sections: Name: foo Version: 1.0 License: BSD3 Cabal-Version: >= 1.9.2 Build-Type: Simple Benchmark bench-foo type: stdio-1.0 main-is: bench-foo.hs build-depends: base This doesn't add much on top of normal executable sections, except that it tells Cabal that this isn't an executable it should build by default or install. Step 2. Add a detailed benchmark interface that allows the benchmark to associate a real valued *measure* (optionally with upper and lower bounds) with a benchmark name. Benchmark naming would work like test naming (e.g. group/subgroup/name). This would allow us to track performance over time. Step 2 needs to be fleshed out some more, but I think we can get quite a lot of bang-for-the-buck with step 1. Cheers, Johan