Re: [Haskell-cafe] Re: GSoC: Improving Cabal's Test Support

On Wed, 2010-04-07 at 16:09 -0400, Thomas Tuegel wrote:

On Wed, Apr 7, 2010 at 3:33 PM, Duncan Coutts wrote:

...

The importance of this is that it lets us develop improved testsuite interfaces in future. At the moment there are two test interfaces we want to support. One is the simple unix style "exit code + stdout" interface. This is good because it is a lowest common denominator that all existing testsuites can fit into.

Of course that test interface does not provide any detailed machine-readable information (though you do get human-readable test logs). So that's why we want a second interface. That one should let the "testing agent" (for example "cabal test" but could be other agents) get much more detail about what tests can be run and then what the results are of various tests.

For the purpose of differentiating between these two, would a field in the test section such as "interface: stdout" (in the first case) or "interface: detailed" (in the second) suffice?

Yep.

...

The details of such an interface are up for discussion. I do not mind if that is a command line executable interface or a library interface.

That's something I've been thinking about. The former seems more portable. Maybe cabal could call the test program with "test_program --list" to produce a list of tests and "test_program test1,test2,test3" to run some tests.

Having some ideas is good. The details of the interface can be worked out during the project.

I also want to ask how strictly this is within the scope of the SoC project, i.e. how well will my proposal be received if it focuses primarily on the first part of the problem (getting everything working for the stdout interface)? I ask because the detailed interface seems like a much larger mandate given that cabal doesn't really support any of the syntax/features for the simple stdout interface.

Certainly we want to get the first part working first. My guess is that there is time within the 3-month GSoC period to complete the basic bits and to at least have a go at a prototype of a more detailed interface. That part doesn't need to be final, especially given that it will be possible to create new interfaces in future.

...

...

Handling of Test Executables by Cabal

The changes proposed here will make it possible to build, test, and install a Cabal package with the usual sequence of commands:

That all sounds reasonable. I'd like to add that the interface between the testsuite and a testing agent such as the cabal program should be clearly documented and specified. There will likely be dedicated test agents that want to run the tests too and send reports to other systems (e.g. dedicated hackage test agents) and convert to other formats (e.g. integration in companies in-house systems).

Rogan mentioned possible upcoming support in test-framework for JUnit XML as an output format for test results. That certainly seems to be widely supported; do you think it is suitable?

I think it's important to be able to convert into standard or custom formats. I've no idea if JUnit XML would make sense as the native format. It's plausible. Duncan