On 12 November 2010 01:16, Thomas Tuegel
First, let me apologize for taking so long to respond; school is keeping me quite busy.
No problem.
* Separation of pure/impure tests - what do you want to use this for? Good unit tests should be isolated from each other (no dependencies). Given this constraint, we can safely run IO-type unit tests in parallel with each other and so on - so it seems that knowing they are pure is not of great benefit?
Good unit tests _should_ be isolated, but I'm not sure we can always count on good ones. Truly pure tests can't help but be isolated, though. You are right that the majority of the time, it won't matter, but I think the overhead of keeping the distinction around is low enough to be worth it.
I still don't think it's worth it. There may be a valuable distinction to make between "tests safe for running concurrently with other tests" and "tests unsafe for running concurrently with other tests". It is true that: "Test is pure" ==> "Test safe for running concurrently" But the reverse direction of the implication is not true: some impure tests are safe for running concurrently. So impure/pure is the wrong distinction: it lumps safe impure tests in with unsafe impure tests. I would prefer that: 1) If we care about supporting unsafe tests, we provide a flag or SafeTestable/UnsafeTestable type class split 2) Even better, we could not support unsafe tests at all. They should be extremely uncommon, and if the user really wants to do that they can implement it themselves by wrapping all tests in a multiple-reader single-writer lock: have safe tests aquire a reader lock and unsafe ones the writer lock during execution. This strategy can even be implemented by combinator of a type such as :: [(Safe, Test)] -> IO [Test] (where type Safe = Bool) outside of the cabal test library, so that the user wouldn't need to write the locks themselves.
I also want to discuss what we're going to do with the TestOptions type. The criticisms it received at HIW all centered around the "type-class as dictionary" pattern that so many Haskellers find distasteful. In my own opinion, the interface would be awkward to use and didn't provide adequate protections against setting invalid options. I want to propose the following interface instead of what currently exists:
OK, so what are we actually trying to achieve with TestOptions? As I see it, the reason we are exposing this information at all (as opposed to just having run/runM consume a [String]) is so that we can do command line argument parsing on behalf of what I call the test "provider". We could also use the infrastructure to populate a GUI with a list of test provider options, and generate usage information. If this is our goal, we should probably choose a different API, closer to that provided by GetOpt or similar. The proposed API only supports arguments of the form --foo=bar, where bar is a string. There are many other common patterns: * --baz (with no argument) * -v vs. --verbose (short form arguments) * --meh={a,b,c} (arguments whose values are drawn from an enumeration -- you could imagine e.g. populating a combo box with these) We are unlikely to do a good job at covering all use cases here. We could: 1. Accept that, and just stick with only providing --foo=bar arguments 2. Support more forms of arguments by using the GetOpt Option data type or our own implementation of it 3. Just delegate everything to the test provider by having them provide a (run :: p -> [String] -> IO (Either String Result)), where errors in the arguments are reported with Left "Error information". Optionally we could add a (usage :: IO String) function for retrieving usage information for the command line. My preferred option is 3. as it is the lightest weight solution, and doesn't require auxiliary OptionField/TestOptions data types. What do you think? Do we lose something important by this choice? Cheers, Max