On Thu, Apr 1, 2010 at 9:36 PM, Ivan Lazar Miljenovic
I propose to build a test suite as its own executable, but to avoid the problem of granularity by producing an output file detailing the success or failure of individual tests and any relevant error messages. The format of the file would be standardized through library routines I propose to write; these routines would run tests with HUnit or QuickCheck and process them into a common format. Cabal, or any other utility, could read this file to determine the state of the test suite. Perhaps Cabal could even warn the user about installing packages with failing tests.
All well and good, but your sample code (which I've ommitted for the sake of brevity) doesn't seem to lead to much room for customisation: for graphviz's test suite, I include various comments about the purpose of the test, etc. as well as changing some of QuickCheck's paramaters; does your proposal allow the inclusion of such customisations?
I've been looking at graphviz's test suite. You used a type
data Test = Test { name :: String , desc :: String , test :: IO Result -- ^ QuickCheck test. }
to represent tests (sorry for mangling the indentation), but this is practically the same as the type I suggested:
type Test = (String, IO (Bool, String))
The String output in the IO monad can just as easily be used to output a test description for a successful test as it can be used for an error message for a failing test. Other than that, the only difference is that you keep the entire QuickCheck Result, and I proposed just to keep a Bool (for compatibility between QuickCheck and HUnit). The use of Bool is purely illustrative; if I use one of the frameworks I've been discussing with Rogan, it will be whatever that framework uses, instead of a Bool.
As an aside, I question the necessity of this kind of proposal: how many people actually run tests for packages they download from Hackage? graphviz's test suite runs for 110 minutes, and I mainly use it to test that my changes/inclusions in what it prints and parses is consistent: I don't expect a user to bother running it (and would question why anyone would).
It's true that this proposal does much more for package authors than for users. I'm not proposing to require users to run tests for every package. At the same time, I'm sure that on the Linux side, distribution packagers would appreciate wider and more uniform availability of package tests; this proposal would help on both fronts.
How does the inclusion of a "test" option to Cabal allow any substantial benefits to developers over building the package with a build-time flag to enable building a test suite?
Admittedly, you stand to benefit less than a package author without such a complete test suite. At the very least, the proposed 'Test' stanza for the package description file would make it unnecessary for you to worry about recognizing a build-time flag and messing about with UserHooks. (As I said to Rogan, using the 'Test' stanza wouldn't lock you in to a particular framework: you could use it to compile your test suite as-is.) That lowers the barrier to entry for creating test suites so more authors will use them, and that's a substantial benefit. I also envision automated testing being offered as a service for package authors: I would like, e.g., to ensure that my package works correctly on different architectures and platforms, but I don't have the means to acquire all those different machines. Collectively, however, the community does. If we used a standard format for handling test results, we could offer a service where uploaded packages would be compiled and tested on an array of machines. Since we're using a common format, we can write one tool to parse and summarize the results across all those platforms, and authors can immediately see where their packages are failing. Certainly, any single package author could write a tool to do this for any single package's tests, but now we have a common tool everyone can use simply by writing tests. To summarize: everyone's life is easier if we do this bit of plumbing for them. -- Thomas Tuegel