On Wed, Apr 28, 2010 at 8:19 AM, Duncan Coutts
Yes, it means the testing agent (cabal-install or some other program/system) can do more than simply run all the tests. It means it can enumerate them and not run them (think a GUI or web interface), run a subset of tests, run them in parallel etc.
I'm not convinced that this should be cabal's responsibility. I think we would be better served by leaving this up to the test frameworks (indeed, test-framework has test filtering capabilities already). If 'cabal test' simply acts as a thin layer between the user/invoking system and the test framework, then we could pass arguments through to the underlying test binary and perform these tasks using whatever interface that test binary provides. This will buy us more flexibility in the long run. (I think this is at least a good place to start -- and matches my interpretation of Thomas's proposal.) If Cabal takes on these responsibilities, then the testing api will be more constrained -- we won't be able to experiment with new test formats/methodologies as easily, since any tests will have to meet a specific API. While I agree that we need standardization, I think that we should achieve that by using compatible output formats and compatible (user) interfaces (and enforcing those with tests, schema checkers, etc..). I don't see many benefits to baking this functionality into cabal when it could be done separately. --Rogan