On 21 May 2011 15:20, Duncan Coutts
So we either have a individual test or a group of tests (which themselves can contain groups etc). I'm sure this is nice for displaying in a UI what tests can be run and displaying results of a run. Imagine an interactive UI that lets the user select a group of tests to be run, or an html report showing results by group.
We could do something like this:
data Tests = Test TestInstance | TestGroup TestName [Tests]
Actually, we need more than this. We need to be able to do IO to enumerate the tests in the group. Consider the ghc tests suite. It is an interesting and reasonably large scale example. I think we ought to make sure that our test suite interface enables us to wrap the current ghc testsuite without having to do a major rewrite. The ghc test suite is implemented as a whole bunch of files arranged into directories with a few test-specific scripts in a few places. So it matches the heirarchy notion well enough but to list the tests in each group (in each dir in the ghc testuite case) means doing IO to list the files in the directory and pull out the ones that are recognised as test cases. The ghc example also makes me thing that we want a bit more declarative info that could be presented in a web/gui interface. I think we want optional descriptions on each group of tests, or on individual tests. Many of ghc tests have a notion of "way", that is you can run the same test with normal, ghci, profiled, dynamic etc ways. Is there anything we can do to expose that information? I don't think we want to make individual test instances paramaterised, but perhaps we can do something slightly more detailed than TestGroup to indicate that some test instances are related to each other in certain ways. For the ghc "way" example, suppose we annotated each test with its way. A UI could then allow things like filtering on these attributes. For ghc's testsuite the use case is to let users do things like only run tests the ghci way, or exclude the opt way. In the ghc test suite the current driver does things like grouping failing tests so that the test name is listed only once, and each failing way is listed together, e.g.: T5636 (ghci, opt, prof) So what if in addition to this system of test options (inputs) we had a similar declarative system for describing test attributes. What might it look like and how could test agents expose this so that users can make use of it? Duncan