On 11 October 2010 13:38, Duncan Coutts
But yes, a review at some point would be appreciated. The goal is to have something close to what test-framework has always had (ie tree of tests, QC, HUnit, other tests to be lifted in) but using as few language extensions as we can get away with. I also rather like Thomas' pure/impure distinction. It's also important of course to handle things like RNG seeds so that we can reproduce test runs.
OK, I've done a bit of a review. Features of test-framework not in "cabal test": * Test groups * Ability to see summary of test results by test type (e.g. 2 failed properties, 5 failed test cases) * Incremental generation of test results (so you can e.g. see interactively the # of properties run so far while a property is still executing) Features of "cabal test" not in test-framework: * The result data type lets you distinguish between "errors" and "failures" * More extensible options type (i.e. it *is* actually extensible :-) Questionable features of "cabal test": * I'm not too sure what the "options" member of "TestOptions" gives you - after all, the test has to do it's own parsing of the [(String, String)] Options list, in "check" and "run"/"runM" - so Cabal doesn't really get to use this information * Separation of pure/impure tests - what do you want to use this for? Good unit tests should be isolated from each other (no dependencies). Given this constraint, we can safely run IO-type unit tests in parallel with each other and so on - so it seems that knowing they are pure is not of great benefit? We could take a union of these two feature sets to get the final "cabal test" interface. Alternatively we could take the union of the sets less the "incremental generation" feature of test-framework. This is a rather nice feature for those running tests interactively. However, it is also one of the most technically challenging parts of the code - for a flavour of the stuff it requires please see my "Improving IO" monad at http://github.com/batterseapower/test-framework/blob/master/core/Test/Framew.... The idea of this monad is to model IO actions that are also able to yield information about their continuing execution on a Chan that can be polled by the thread responsible for actually displaying test progress to the user. My current implementation of this feature is the source of lots of all of multiparameter type classes and fundeps in the code, but there is a way to do it in Haskell 98. Thus the only cost of the feature would be (localised) complexity in the code of what you call the "test frameworks" that translate e.g. QC properties into the Test data type, and the "test agents" that actually run the tests. Without the incremental generation feature, we would just need to make this sort of change to the interface: {{{ +data Tests = ATest String Test + | Tests String [Tests] -runTests :: [TestSuite.Test] -> IO () +runTests :: [TestSuite.Tests] -> IO () class TestOptions t where + -- | Return a general description of the sort of test being run, e.g. "Property" or "Test Case" + testTypeName :: t -> String -test :: QC.Testable prop => String -> prop -> Cabal.Test +test :: QC.Testable prop => String -> prop -> Cabal.Tests }}} Should we add the (more invasive) incremental result generation stuff as well? Cheers, Max