Proposal: Add a benchmark section to .cabal files

4 Oct 2011

      Hi all,

Problem
=======

After having used Makefiles to build my benchmark suites for quite some time
I'm convinced we need support for building benchmark using Cabal. Make
doesn't solve the problem of building a benchmarks particularly well. There
are several problems:

* Building your library using Cabal and your benchmarks using Make sometimes
ends up with different package versions being used for the two separate
builds, resulting in compilation errors.
* Lots of functionality that's already in Cabal is duplicated in Makefiles
(e.g. preprocessor rules).
* It's hard to get proper incremental compilation; I always use `make -C
benchmarks clean all` to rebuild the benchmarks, just to make sure that
everything is up-to-date.

Proposal
========

Step 1. Add basic support for building simple benchmark suites e.g. modelled
after test suite sections:

Name:           foo
Version:        1.0
License:        BSD3
Cabal-Version:  >= 1.9.2
Build-Type:     Simple

Benchmark bench-foo
    type:           stdio-1.0
    main-is:        bench-foo.hs
    build-depends:  base

This doesn't add much on top of normal executable sections, except that it
tells Cabal that this isn't an executable it should build by default or
install.

Step 2. Add a detailed benchmark interface that allows the benchmark to
associate a real valued *measure* (optionally with upper and lower bounds)
with a benchmark name. Benchmark naming would work like test naming (e.g.
group/subgroup/name). This would allow us to track performance over time.

Step 2 needs to be fleshed out some more, but I think we can get quite a lot
of bang-for-the-buck with step 1.

Cheers,
Johan

Johan Tibell

Duncan Coutts

Johan Tibell

Antoine Latter

Tillmann Rendel

Duncan Coutts

Johan Tibell

tags

participants (4)