Re: [Haskell-cafe] Re: GSoC: Improving Cabal's Test Support

On Wed, 2010-04-28 at 09:55 -0700, Rogan Creswick wrote:

On Wed, Apr 28, 2010 at 8:19 AM, Duncan Coutts wrote:

...

Yes, it means the testing agent (cabal-install or some other program/system) can do more than simply run all the tests. It means it can enumerate them and not run them (think a GUI or web interface), run a subset of tests, run them in parallel etc.

I'm not convinced that this should be cabal's responsibility.

"Cabal" should define the interface between testsuite and test runner. Nothing more. Packages should provide collections of tests. Testing agents should provide a test runner to actually run the tests and do something with the results. See for example test-framework which has exactly this decomposition between testsuites (a collection of tests) and a test runner. They are mediated by a common interface. The test-framework package provides both the interface, some adapters for QC/HUnit to provide tests, and also a sample test runner that prints results to the console. Tools like cabal-install that use the interface defined by Cabal can provide a test runner (almost certainly implemented in some other package) and then do something interesting with the results like showing them to the user or uploading them to hackage. Other tools can use other test runners and do other interesting things.

I think we would be better served by leaving this up to the test frameworks (indeed, test-framework has test filtering capabilities already). If 'cabal test' simply acts as a thin layer between the user/invoking system and the test framework, then we could pass arguments through to the underlying test binary and perform these tasks using whatever interface that test binary provides. This will buy us more flexibility in the long run. (I think this is at least a good place to start -- and matches my interpretation of Thomas's proposal.)

If Cabal takes on these responsibilities, then the testing api will be more constrained -- we won't be able to experiment with new test formats/methodologies as easily, since any tests will have to meet a specific API.

It is exactly defining this API that should allow flexibility. It means packages can provide tests and have them be used in multiple different ways by different test runners with different purposes and capabilities. If all you can do is run the testsuite and collect the results, that's much more constrained. Note that we're also proposing a lowest-common-denominator testsuite interface that gives all the control to the package author, but means the test runner cannot interpret the results in any interesting way. The main point is to wrap any existing testsuite in a way that allows it to be run automatically and non-iteractively for e.g. hackage testing. Duncan