Re: [Haskell-cafe] Re: GSoC: Improving Cabal's Test Support

On Wed, Apr 28, 2010 at 12:55 PM, Rogan Creswick wrote:

On Wed, Apr 28, 2010 at 8:19 AM, Duncan Coutts wrote:

...

Yes, it means the testing agent (cabal-install or some other program/system) can do more than simply run all the tests. It means it can enumerate them and not run them (think a GUI or web interface), run a subset of tests, run them in parallel etc.

I'm not convinced that this should be cabal's responsibility.

I think we would be better served by leaving this up to the test frameworks (indeed, test-framework has test filtering capabilities already).  If 'cabal test' simply acts as a thin layer between the user/invoking system and the test framework, then we could pass arguments through to the underlying test binary and perform these tasks using whatever interface that test binary provides.  This will buy us more flexibility in the long run.  (I think this is at least a good place to start -- and matches my interpretation of Thomas's proposal.)

That is more or less how I intended my proposal to be read, the caveat being that I intentionally said very little about the detailed test suite interface. I agree that we should leave much up to the testing frameworks. If we start implementing facilities in Cabal to pick and choose specific tests from inside test suites, we're essentially writing yet another test framework into Cabal; I've been specifically discouraged from doing that since I began discussing this proposal on the list. I am increasingly of the opinion that we should just provide a simple, stdout/exit code interface and let test frameworks handle the rest: If developers want continuous integration with existing testing tools, the can use a framework that supports the output format those tools use, and pipe that output to stdout to be captured by Cabal. Then they can turn whichever tool they want loose on the output file. If developers want to independently run subsets of their tests, they can give them independent test stanzas in the .cabal file. Either they put the tests in different executables, or the test framework can provide command-line options for turning tests on and off. Those are the big two usage scenarios we've discussed for the detailed test interface, and I think these examples demonstrate why I think it may be unnecessary. -- Thomas Tuegel