On Sat, May 21, 2011 at 7:36 PM, Duncan Coutts
Actually, we need more than this. We need to be able to do IO to enumerate the tests in the group.
Consider the ghc tests suite. It is an interesting and reasonably large scale example. I think we ought to make sure that our test suite interface enables us to wrap the current ghc testsuite without having to do a major rewrite.
The ghc test suite is implemented as a whole bunch of files arranged into directories with a few test-specific scripts in a few places. So it matches the heirarchy notion well enough but to list the tests in each group (in each dir in the ghc testuite case) means doing IO to list the files in the directory and pull out the ones that are recognised as test cases.
The ghc example also makes me thing that we want a bit more declarative info that could be presented in a web/gui interface. I think we want optional descriptions on each group of tests, or on individual tests. Many of ghc tests have a notion of "way", that is you can run the same test with normal, ghci, profiled, dynamic etc ways.
Is there anything we can do to expose that information? I don't think we want to make individual test instances paramaterised, but perhaps we can do something slightly more detailed than TestGroup to indicate that some test instances are related to each other in certain ways. For the ghc "way" example, suppose we annotated each test with its way. A UI could then allow things like filtering on these attributes. For ghc's testsuite the use case is to let users do things like only run tests the ghci way, or exclude the opt way. In the ghc test suite the current driver does things like grouping failing tests so that the test name is listed only once, and each failing way is listed together, e.g.:
T5636 (ghci, opt, prof)
So what if in addition to this system of test options (inputs) we had a similar declarative system for describing test attributes. What might it look like and how could test agents expose this so that users can make use of it?
I'm worried about generalizing from a single instance (i.e. GHC). If we'd like to add some flexibility we could allow tests to carry arbitrary tags: class TestInstance a where tags :: a -> [String] Test agents could use tags to group tests in different ways. We could add support for tags in Cabal's test runner so that they can be used to select which tests to run. For example, cabal test --test-tags="smoke -flaky" could run all non-flaky smoke tests. Johan