On Wed, Jul 27, 2011 at 5:06 AM, Johan Tibell wrote:

A few notes:

* Calling the data type that contains the Test, Group, and ExtraOptions constructor Tests feels a bit weird, as you often end up having to give a list of these e.g. to the Group constructor. You end up with something of type [Tests]. I think Test is a better name for this type.

It's true; I _do_ feel weird having something with the type [Tests]. I will change it.

* Having the top-level 'tests' binding be IO Tests, instead of IO [Tests], is a bit inconvenient. It forces the users to always use at least one group, which forces him/her to name that group. As we all known, naming things is one of the two hard problems in computer science (the other being cache coherency).

If we do this, should test runners assume that the list of tests can safely be run concurrently, or not? My primary motivation for requiring the user to use a group was the explicit answer to this question.

* I'm a bit confused about how Progress would be used in practice. It returns a String. What is the client supposed to do with this string? Returning a String doesn't seem enough to implement some kind of update-in-place cursors interface.

The String is intended to be a status message, something to display to the user. For comparison, the intermediate types used by test-framework are only required to have a Show instance, so I think this is sufficient.

* Perhaps we should expose some convince functions to create Test values. In particular I feel like you'd rarely want to create a TestInstance value without wrapping it in a Test constructor. I like the testGroup function exported by test-framework.

I will have a look at the convenience functions test-framework offers, and see which would be useful here. Thanks! -- Thomas Tuegel

On Thu, 2011-07-28 at 21:25 -0500, Thomas Tuegel wrote:

On Wed, Jul 27, 2011 at 7:32 PM, Duncan Coutts wrote:

...

Speaking of IO, do we ever need to deal with test suites that need imperative setup and teardown for sets of tests? Individual tests can do setup and teardown via the Progress which embeds IO. Does HUnit allow IO for sets/groups of tests?

I can see the utility of this. It would be simple enough to have Group take an (IO [Tests]) instead of just [Tests].

Note that that's not quite the same thing. That lets you do IO to enumerate the tests, not IO before and after running a group of tests. I'm slightly confused about what you and Johan are talking about with [Tests] etc. Can you post what the new proposed data types are please? As for convenience functions, I don't know if it's worth it. We're not expecting packages to implement this directly. It'll just be for packages like test-framework to implement. But if they're cheap and obvious then I don't object strongly. Duncan

On Tue, 2011-08-16 at 14:23 -0500, Thomas Tuegel wrote:

Johan and I are talking about renaming the Tests type to just "Test" to be more consistent about pluralization. So, also incorporating the changes I think you are suggesting to the Group type, it would look like:

data Test = Test TestInstance | Group { groupName :: String , concurrently :: Bool , groupTests :: [Test] , preGroup :: IO () , postGroup :: IO () } | ExtraOptions [OptionDescr] Test

I think leave off the pre/post IO for now. It's not going to be really usable like that. It'd have to be something horrible like: data Test = Test (TestInstance a) | forall a. Group { groupName :: String , concurrently :: Bool , groupTests :: [Test a] , preGroup :: IO a , postGroup :: a -> IO () } data TestInstance a = TestInstance { run :: a -> IO Progress since the individual tests within a group need a handle to whatever was done in the pre/setup phase. For example imagine a testsuite for HDBC. Testing the backends requires creating a connection to a database, populating it with a bunch of test data, running one or more tests on that test db, and then cleaning and closing the db at the end of the tests. It's tricky because we want to enumerate all the tests without running setup code, but to run a test we'd have had to do all the setup and pass any results in to the tests. So I don't suggest we tackle this in full right now. We have a system with versioned test suite interfaces for a reason, and that's so we can change and add stuff later. Everything else seems fine. Lets try it out. Duncan

On Wed, 2011-08-31 at 09:30 +0200, José Pedro Magalhães wrote:

Hi all,

I tried to use the detailed-1.0 test interface, according to the instructions on http://www.haskell.org/cabal/users-guide/#test-suites.

Ah, a rare case of premature documentation. That is, documentation for a feature that is due, but not released yet. I'll update the docs to hide the mention of the detailed interface for the moment.

What am I doing wrong?

Nothing.

It seems like my cabal does not support anything other than exitcode...

That's right. No released version supports the detailed interface yet. Just recently we think we've settled on the final interface and will hopefully get that implemented in the darcs version soon. If you'd like to be guinea pig for that then I expect Thomas would like to talk to you. Duncan

On Tue, 2011-09-06 at 11:16 +0200, José Pedro Magalhães wrote:

Hi Duncan,

Is there a planned date for a release which will have the detailed test interface?

My hope is that it'll be ready in time to be included in a release of Cabal to go along with GHC 7.4 towards the end of the year. If you'd like to help move things along then you could help us try out the new proposed interface. Thomas posted the latest patches yesterday. Duncan

Greetings once again! I have attached an updated patch with the requested changes to the test interface. (I apologize for the delay; it was caused by the unfortunate conspiracy of moving, not having internet access, and qualifying exams. All of these being done, I will be more responsive on Cabal-related things.) Here is a summary of the patches, with some commentary: This is the patch from before: Tue Jul 26 17:29:23 CDT 2011 Thomas Tuegel * New detailed test suite interface. This patch implements a new interface for detailed test suites based on Duncan's proposal. This implementation differs from his in a few ways: * The constructors of 'Tests' have been renamed. I think 'TestGroup' and 'ExtraTestOptions' are redundant: it is clear from context what sort of Group and what sort of ExtraOptions are being considered and qualified imports can resolve any name conflicts. Group and ExtraOptions have the advantage of improving the brevity of pattern matches on Tests, which are used very often in D.S.Test. * The 'concurrentSafe :: Bool' field of TestInstance has become the 'concurrently' field of Group, allowing package and test framework authors greater control over concurrency. * The 'Finished' constructor of 'Progress' now contains the options used to run the test in addition to the test result. Without returning the options, it would be difficult to extract the RNG seed used to run a test. * A detailed test suite module now exports the symbol 'test :: IO Tests'. This enables the use of IO to enumerate the tests in a group, suggested by Duncan as a way to accomodate the GHC test suite. The name change Johan suggested: Fri Jul 29 10:02:38 CDT 2011 Thomas Tuegel * Renamed Distribution.TestSuite.Tests to Test. Forgot to remove ExistentialQuantification. Fri Jul 29 10:10:01 CDT 2011 Thomas Tuegel * Removed obsolete LANGUAGE pragma from Distribution.TestSuite. Fri Jul 29 10:42:49 CDT 2011 Thomas Tuegel * Cleaned for warnings in D.TestSuite and D.S.Test. Duncan pointed out that this field is redundant: Fri Jul 29 10:48:12 CDT 2011 Thomas Tuegel * Removed 'optionStringDescription' from OptionType. Johan suggested having the symbol exported by test modules be "tests :: IO [Test]" rather than "tests :: IO Test". Formerly, all the tests in a suite would have to be wrapped in a named Group, and as Johan pointed out, naming things is a hard problem. Thu Sep 1 13:26:06 CDT 2011 Thomas Tuegel * Changed detailed test exported type to [Test]. After the last patch, the test cases in a suite aren't structured as a simple tree, so the 'OptionTree' type I devised for reading options from a log file and reapplying them doesn't make much sense. I opted to remove it for the reasons explained below. Tue Sep 6 16:16:11 CDT 2011 Thomas Tuegel * Clean D.S.Test for unused symbols. The unused declarations were all related to replaying test suites with logged options, but the command-line option for this feature has been disabled for some time. Changing the detailed test suite type to expect "tests :: IO [Test]" instead of "tests :: IO Test" made the old method of replaying options clumsy. Since it was already disabled, I chose to remove it, rather than rewrite it again. Per Duncan's suggestion: Tue Sep 6 16:55:57 CDT 2011 Thomas Tuegel * Removed Options from Finished. The documentation makes explicit the fact that the 'concurrently' property is not inherited. Tue Sep 6 16:56:00 CDT 2011 Thomas Tuegel * Improved documentation for 'concurrently' field of Test. The convenience function Johan requested: Tue Sep 6 18:36:32 CDT 2011 Thomas Tuegel * Added 'testGroup' to D.S.TestSuite. Since most of the major changes are done, I updated the user manual (so it won't fall through the cracks later). Tue Sep 6 18:49:31 CDT 2011 Thomas Tuegel * Updated user manual for new detailed interface. There's still the issue of a trial implementation of the interface: On Wed, Jul 27, 2011 at 7:32 PM, Duncan Coutts wrote:

Perhaps one of us should have a go at seeing if we can make the ghc test suite fit this interface. Perhaps a project for the hackathon...

I had a look at the GHC test suite. It looks to me like the Python test runner would have to be replaced with some Haskell code to make it fit the detailed interface. I don't have any experience with GHC or its test suite, though; there may be an easier way that I'm not seeing. -- Thomas Tuegel