On Mon, May 23, 2011 at 03:48:05PM +0200, Johan Tibell wrote:
On Sat, May 21, 2011 at 7:36 PM, Duncan Coutts
T5636 (ghci, opt, prof)
So what if in addition to this system of test options (inputs) we had a similar declarative system for describing test attributes. What might it look like and how could test agents expose this so that users can make use of it?
I'm worried about generalizing from a single instance (i.e. GHC).
If we'd like to add some flexibility we could allow tests to carry arbitrary tags:
class TestInstance a where tags :: a -> [String]
Just because GHC's testsuite distinguishes between ways and tests, I'm not sure if Cabal's testsuite framework needs to do so. Ways could just be encoded into the hierarchy, e.g. GHC's testsuite could include the tests typechecker.T5636.ghci typechecker.T5636.opt typechecker.T5636.prof rather than the test typechecker.T5636 with 3 tagged variants. Perhaps having tags /as well/ would make sense, so typechecker.T5636.ghci would be tagged ["typechecker", "ghci"] and it would then be possible to run 'all tests tagged "ghci"' or 'all tests tagged "typechecker"'. Thanks Ian