Re: [Haskell-cafe] Re: GSoC: Improving Cabal's Test Support

On Apr 6, 2010, at 5:28 PM, Thomas Tuegel wrote:

On Tue, Apr 6, 2010 at 7:43 PM, Rogan Creswick wrote:

...

test-framework and test-runner both address the second problem, and those solutions can be kept separate, at least for now. Figuring out the best way to specify test commands, dependencies, build/execution order, etc. is going to take some substantial effort, and I think that should be the first goal of the project.

Ok, this is the bottom-line that I didn't understand after our first exchange, but I think now I do: I should entirely scrap the second aspect of my proposal and focus exclusively on making Cabal build and run test programs.

I concur with this conclusion.

...

Cabal allows for multiple executable sections -- are multiple test sections allowed? If so, how are they handled when 'cabal test' is invoked? If not, will there be any support for running multiple test suites? (more on this below).

While the test executable could be configured to run different sets of tests (at runtime? hm.. we may need more flags to 'cabal test'), there are some situations it's necessary to build multiple test suites because of odd library dependencies. (for example, testing certain combinations of libraries--imagine supporting multiple versions of ghc.)

I had intended to allow multiple 'Executable' sections; that's what I meant by "this is really an 'Executable' stanza by another name". I think that takes care of your concern about building multiple test suites with different dependencies, also.

My thinking is that 'cabal test' should run all the tests that were built, and 'cabal test foo' should run only the test named 'foo'.

This sounds like a good idea to me.

...

The existing Executable sections may serve the need fine, if we could specify how to run the tests in a different way. Perhaps a list of test commands could be specified instead, eg:

...

TestCommands: foo-test-ghc6.6, foo-test-ghc6.8, foo-props --all

Anyhow, just food for thought.

One of the reasons I prefer implementing a dedicated 'Test' stanza is that it makes it easier to tell which executables to install. There may be situations where we want to install test programs, but there will _always_ be situations where we _don't_ want to. (Maybe '--{en,dis}able-tests' passed to 'cabal install' should control this?) Could we take a 'TestCommands' list and parse out the options to get the executable names? Yes, but relying on that to work makes me uneasy. I think the more robust way to specify executable-specific options instead would be to add a field to the 'Test' section ('run-options' seems to be consistent with the naming scheme) that doesn't exist in 'Executable'.

(I've snipped some of the comments here; if forget about test frameworks for this proposal, it's all tangential.)

I agree with your reasoning here about having a separate Test stanza so that tests can be kept separate from the rest of the package. - Greg