On Tue, Apr 6, 2010 at 7:43 PM, Rogan Creswick
test-framework and test-runner both address the second problem, and those solutions can be kept separate, at least for now. Figuring out the best way to specify test commands, dependencies, build/execution order, etc. is going to take some substantial effort, and I think that should be the first goal of the project.
Ok, this is the bottom-line that I didn't understand after our first exchange, but I think now I do: I should entirely scrap the second aspect of my proposal and focus exclusively on making Cabal build and run test programs.
Cabal allows for multiple executable sections -- are multiple test sections allowed? If so, how are they handled when 'cabal test' is invoked? If not, will there be any support for running multiple test suites? (more on this below).
While the test executable could be configured to run different sets of tests (at runtime? hm.. we may need more flags to 'cabal test'), there are some situations it's necessary to build multiple test suites because of odd library dependencies. (for example, testing certain combinations of libraries--imagine supporting multiple versions of ghc.)
I had intended to allow multiple 'Executable' sections; that's what I meant by "this is really an 'Executable' stanza by another name". I think that takes care of your concern about building multiple test suites with different dependencies, also. My thinking is that 'cabal test' should run all the tests that were built, and 'cabal test foo' should run only the test named 'foo'.
The existing Executable sections may serve the need fine, if we could specify how to run the tests in a different way. Perhaps a list of test commands could be specified instead, eg:
TestCommands: foo-test-ghc6.6, foo-test-ghc6.8, foo-props --all
Anyhow, just food for thought.
One of the reasons I prefer implementing a dedicated 'Test' stanza is that it makes it easier to tell which executables to install. There may be situations where we want to install test programs, but there will _always_ be situations where we _don't_ want to. (Maybe '--{en,dis}able-tests' passed to 'cabal install' should control this?) Could we take a 'TestCommands' list and parse out the options to get the executable names? Yes, but relying on that to work makes me uneasy. I think the more robust way to specify executable-specific options instead would be to add a field to the 'Test' section ('run-options' seems to be consistent with the naming scheme) that doesn't exist in 'Executable'. (I've snipped some of the comments here; if forget about test frameworks for this proposal, it's all tangential.)
See my comments about running multiple test suites, and parameterized test suites above. I think richer parameters are necessary. (possibly just a --pass-through flag that hands all the subsequent parameters off to the test executable(s))
That is an excellent suggestion that I will definitely adopt in my eventual proposal. Thanks! -- Thomas Tuegel