First, let me apologize for taking so long to respond; school is keeping me quite busy. On Mon, Oct 11, 2010 at 5:13 PM, Max Bolingbroke wrote:

OK, I've done a bit of a review.

Features of test-framework not in "cabal test":  * Test groups  * Ability to see summary of test results by test type (e.g. 2 failed properties, 5 failed test cases)

I agree that these features should be in cabal test.

 * Incremental generation of test results (so you can e.g. see interactively the # of properties run so far while a property is still executing)

I think this is a good idea. It is surely worth the complication, but I don't think of it as a top priority, at least until some other issues are resolved.

Features of "cabal test" not in test-framework:  * The result data type lets you distinguish between "errors" and "failures"  * More extensible options type (i.e. it *is* actually extensible :-)

Questionable features of "cabal test":  * I'm not too sure what the "options" member of "TestOptions" gives you - after all, the test has   to do it's own parsing of the [(String, String)] Options list, in "check" and "run"/"runM" - so Cabal   doesn't really get to use this information

I want to rework TestOptions anyway, based on my own distaste for it and the several criticisms it received at the Implementors Workshop. (More on this below.)

 * Separation of pure/impure tests - what do you want to use this for? Good unit tests should be isolated from each other (no dependencies). Given this constraint, we can safely run IO-type unit tests in parallel with each other and so on - so it seems that knowing they are pure is not of great benefit?

Good unit tests _should_ be isolated, but I'm not sure we can always count on good ones. Truly pure tests can't help but be isolated, though. You are right that the majority of the time, it won't matter, but I think the overhead of keeping the distinction around is low enough to be worth it. I also want to discuss what we're going to do with the TestOptions type. The criticisms it received at HIW all centered around the "type-class as dictionary" pattern that so many Haskellers find distasteful. In my own opinion, the interface would be awkward to use and didn't provide adequate protections against setting invalid options. I want to propose the following interface instead of what currently exists:

-- | Abstraction of a field for an internal type. data OptionField i = OptionField { -- | The name of the field. Used to construct the 'TestOptions'. name :: String -- | The type of the field. Used to construct the 'TestOptions'. , fieldType :: TypeRep -- | Read and set this field in the internal type, returning the new data if the input was valid. , setter :: i -> String -> Maybe i -- | Show this field from the internal type. , getter :: i -> String }

data TestOptions p = TestOptions { -- | Set the field named in the first argument to the value -- read from the second. Return 'Nothing' if the field does -- not exist or the value cannot be read. set :: String -> String -> Maybe (TestOptions p) -- | Return the value of the named field, if it exists. , get :: String -> Maybe String , fields :: [(String, TypeRep)] }

mkTestOptions :: [OptionField i] -> i -> TestOptions p

class Testable t where defaultOptions :: IO (TestOptions t) testTypeName :: t -> String testName :: t -> String

class Testable p => PureTestable p where run :: p -> TestOptions p -> Result

class Testable i => ImpureTestable i where runM :: i -> TestOptions i -> Result

If we agree that this would be an improvement, I will submit a patch. -- Thomas Tuegel

Hello! Max has some excellent suggestions for detailed test interface: The first subject we've been discussing is the orthogonality of the pure/impure distinction from safe/unsafe. Specifically, the current interface makes the pure/impure distinction, but we really care about the safe/unsafe distinction. There are, e.g., safe, impure tests. One thing to do would be to change the interface so there is only one "Testable" class, allow tests in it to run in IO, and provide some kind of combinator for denoting safe/unsafe tests. On Mon, Nov 15, 2010 at 6:47 AM, Max Bolingbroke wrote:

2) Even better, we could not support unsafe tests at all. They should be extremely uncommon, and if the user really wants to do that they can implement it themselves by wrapping all tests in a multiple-reader single-writer lock: have safe tests aquire a reader lock and unsafe ones the writer lock during execution. This strategy can even be implemented by combinator of a type such as :: [(Safe, Test)] -> IO [Test] (where type Safe = Bool) outside of the cabal test library, so that the user wouldn't need to write the locks themselves.

One other thing we've been discussing is what to do about TestOptions. The fallout of this discussion is essentially that everything TestOptions is trying to do, GetOpt does better. On Mon, Nov 15, 2010 at 6:47 AM, Max Bolingbroke wrote:

If this is our goal, we should probably choose a different API, closer to that provided by GetOpt or similar. The proposed API only supports arguments of the form --foo=bar, where bar is a string. There are many other common patterns: * --baz (with no argument) * -v vs. --verbose (short form arguments) * --meh={a,b,c} (arguments whose values are drawn from an enumeration -- you could imagine e.g. populating a combo box with these)

We are unlikely to do a good job at covering all use cases here. We could: 1. Accept that, and just stick with only providing --foo=bar arguments 2. Support more forms of arguments by using the GetOpt Option data type or our own implementation of it 3. Just delegate everything to the test provider by having them provide a (run :: p -> [String] -> IO (Either String Result)), where errors in the arguments are reported with Left "Error information". Optionally we could add a (usage :: IO String) function for retrieving usage information for the command line.

Option 1 is what we're unsatisfactorily doing now. Option 2 would be to have a method such as "options :: [OptDescr (t -> t)]" in class Testable. Option 3 is to delegate option parsing to the test provider. I prefer option 2: compared to option 3, it enforces standardization more rigidly and takes some of the work off test provider authors. Thoughts? I thought if we could come to a consensus here, I can have some patches by the end of the week. Thanks! -- Thomas Tuegel

Here, at last, is the patch I promised. It would not have taken this long, but I wanted to get the module documentation correct. So, this patch not only has a new interface, but an example implementation (in the module docs) of the detailed test interface for QuickCheck >= 2.3. The patch does what we talked about: a single "Testable" class and options with GetOpt (the one bundled with Cabal). The interface in this patch assumes that all tests are safe; supporting safe and unsafe tests would be a simple matter of adding another constructor to the "Test" type. The new interface has the advantage of being substantially simpler, i.e., shorter, and without all the exceptions junk. I did not bump the version number for the test type; it's still "detailed-0.9" because I wasn't sure if we want to bump it or what we want to bump it to. (Are we ready to bump to "detailed-1.0"?) On Tue, Jan 18, 2011 at 2:59 PM, Max Bolingbroke wrote:

On 11 January 2011 18:09, Thomas Tuegel wrote:

...

One thing to do would be to change the interface so there is only one "Testable" class, allow tests in it to run in IO,

I would be happy with this.

...

and provide some kind of combinator for denoting safe/unsafe tests.

That might be a possible extension. I am of the opinion that "unsafe" tests (i.e. those which cannot be run in parallel with other tests) are both rare and undesirable from a design perspective, and so we should not provide explicit support for them.

...

One other thing we've been discussing is what to do about TestOptions. The fallout of this discussion is essentially that everything TestOptions is trying to do, GetOpt does better. ... have a method such as "options :: [OptDescr (t -> t)]" in class Testable. Option 3 is to delegate option parsing to the test provider. I prefer option 2: compared to option 3, it enforces standardization more rigidly and takes some of the work off test provider authors.

Option 2 seems like a reasonable choice. GetOpt isn't perfect (I like cmdargs) but it does have the considerable advantage of being a de-facto standard and is in the base package. I think your idea to use [OptDescr (t -> t)] is particularly elegant!

I look forward to seeing your patch :-). Let me (and the list) know if any more issues come up.

Cheers, Max

-- Thomas Tuegel