First, let me apologize for taking so long to respond; school is
keeping me quite busy.
On Mon, Oct 11, 2010 at 5:13 PM, Max Bolingbroke
OK, I've done a bit of a review.
Features of test-framework not in "cabal test": * Test groups * Ability to see summary of test results by test type (e.g. 2 failed properties, 5 failed test cases)
I agree that these features should be in cabal test.
* Incremental generation of test results (so you can e.g. see interactively the # of properties run so far while a property is still executing)
I think this is a good idea. It is surely worth the complication, but I don't think of it as a top priority, at least until some other issues are resolved.
Features of "cabal test" not in test-framework: * The result data type lets you distinguish between "errors" and "failures" * More extensible options type (i.e. it *is* actually extensible :-)
Questionable features of "cabal test": * I'm not too sure what the "options" member of "TestOptions" gives you - after all, the test has to do it's own parsing of the [(String, String)] Options list, in "check" and "run"/"runM" - so Cabal doesn't really get to use this information
I want to rework TestOptions anyway, based on my own distaste for it and the several criticisms it received at the Implementors Workshop. (More on this below.)
* Separation of pure/impure tests - what do you want to use this for? Good unit tests should be isolated from each other (no dependencies). Given this constraint, we can safely run IO-type unit tests in parallel with each other and so on - so it seems that knowing they are pure is not of great benefit?
Good unit tests _should_ be isolated, but I'm not sure we can always count on good ones. Truly pure tests can't help but be isolated, though. You are right that the majority of the time, it won't matter, but I think the overhead of keeping the distinction around is low enough to be worth it. I also want to discuss what we're going to do with the TestOptions type. The criticisms it received at HIW all centered around the "type-class as dictionary" pattern that so many Haskellers find distasteful. In my own opinion, the interface would be awkward to use and didn't provide adequate protections against setting invalid options. I want to propose the following interface instead of what currently exists:
-- | Abstraction of a field for an internal type. data OptionField i = OptionField { -- | The name of the field. Used to construct the 'TestOptions'. name :: String -- | The type of the field. Used to construct the 'TestOptions'. , fieldType :: TypeRep -- | Read and set this field in the internal type, returning the new data if the input was valid. , setter :: i -> String -> Maybe i -- | Show this field from the internal type. , getter :: i -> String }
data TestOptions p = TestOptions { -- | Set the field named in the first argument to the value -- read from the second. Return 'Nothing' if the field does -- not exist or the value cannot be read. set :: String -> String -> Maybe (TestOptions p) -- | Return the value of the named field, if it exists. , get :: String -> Maybe String , fields :: [(String, TypeRep)] }
mkTestOptions :: [OptionField i] -> i -> TestOptions p
class Testable t where defaultOptions :: IO (TestOptions t) testTypeName :: t -> String testName :: t -> String
class Testable p => PureTestable p where run :: p -> TestOptions p -> Result
class Testable i => ImpureTestable i where runM :: i -> TestOptions i -> Result
If we agree that this would be an improvement, I will submit a patch. -- Thomas Tuegel