Rather that starting from scratch, you should strongly consider adapting something like test-framework to this task, as it already has done the heavy work of creating a way to combine tests from different frameworks into a single suite and includes such features as displaying a progress bar during the QuickCheck tests. Furthermore, it is easily extendable to support new kinds of tests; for example, I found that it was relatively straightforward to add a new kind of "statistical" test to make sure that the average value of a function where where it should be. Cheers, Greg On Apr 6, 2010, at 3:51 PM, Thomas Tuegel wrote:
Hello again!
Based on the invaluable feedback I've received, I've made some revisions to the proposal I made a few days ago (at the end of this post, after my signature). I apologize for the length of my post, but I'd like once again to solicit feedback on this. Any commentary is very helpful!
Thanks! -- Thomas Tuegel
Throughout this proposal, examples are given to indicate how a package author would utilize the features proposed here. In all these examples, suppose that the programmer is the author of the 'haskell-foo' package, which exposes the module 'Foo' and has a single test executable, 'foo-tests', using the QuickCheck testing library.
Package Description File Syntax
The syntax for designating test executables in package description files will be based on the existing syntax for describing executables. Such a stanza in the hypothetical package's description file would look like:
Test foo-tests main-is: foo-tests.hs build-depends: haskell-foo, Cabal, QuickCheck
This example is obviously minimal; this is really an 'Executable' stanza by another name, so any options recognized there would also be valid here.
Handling of Test Executables by Cabal
The changes proposed here will make it possible to build, test, and install a Cabal package with the usual sequence of commands:
$ cabal configure $ cabal build $ cabal test $ cabal install
Cabal will recognize two new options during the 'configure' stage: '--enable-tests' and '--disable-tests'.
If 'cabal configure' is invoked with the '--enable-tests' option, then any test executables designated in the package description file will be built. For the purposes of the 'configure' and 'build' stages, they will be handled as if they were ordinary executables, i.e., described by 'Executable' stanzas. With tests enabled, the test programs will be executed and their results collected by Cabal during the 'test' stage.
If 'cabal configure' is invoked with the '--disable-tests' option (which should be the default if neither option is specified), then test executables designated in the package description file will be ignored, as if the 'Test' stanza were absent. Any attempt to invoke the 'test' stage with tests disabled should remind the user of that fact.
Regardless of the status of tests (enabled or disabled), the 'install' stage will ignore any executables designated as test suites, since it is not desirable to install the test executables.
Collection of Test Results
Cabal will provide a standard interface, residing in the module 'Distribution.Test', for running tests independent of the testing library used. A minimal outline of this module looks like:
module Distribution.Test where
type Name = String type Result = Maybe Bool type Info = String type Output = String
-- 'Compiler' and 'ComponentLocalBuildInfo' are already provided by Cabal. -- They are included here to aid in debugging test failures type Report = (Compiler, ComponentLocalBuildInfo, [(Name, Result, Info, Output)])
class Test t where wrap :: t -> IO (Result, Info)
runTests :: Test t => [(Name, t)] -> IO Report
writeResults :: Report -> IO ()
Instances of 'Test' will run a type of test from one of the testing libraries; part of this project will therefore be patching QuickCheck and HUnit to provide these instances. Any other testing library providing this instance will also be compatible with the automated testing features this proposal introduces.
The type 'Maybe Bool' is used throughout this framework to indicate a test result: 'Nothing' indicates a test was not run, 'Just False' indicates a failed test, and 'Just True' indicates a successful test. The 'Info' string captures any information provided by the testing library. However, because of the reliance of most test suites on standard output, Cabal will also capture the standard output produced during each test (when the test suite is invoked through 'cabal test'); the output will be included in the test result file.
The function 'writeResults' will write the test results to a file. The 'Show' instance for the type of its single argument will therefore constitute the standard test result file format. This has the advantage of being human- and machine-readable without requiring any extra dependencies to parse the file.
With this framework, the hypothetical package's author might write a test suite such as:
module Main where
import Distribution.Test import Foo import QuickCheck
testBar :: Gen Bool testBar = ...
testBaz :: Gen Bool testBaz = ...
main = runTests [("testBar", testBar), ("testBaz", testBaz)] >>= writeResults
Reporting and Comparing Test Results
The 'cabal test' command will run tests by default, but support two other options:
1. '--report [file]', which will produce a nicely formatted report of the test results stored in the named file, or of the last run of the package's test suite if no file is specified, and 2. '--diff file1 file2', which will show the differences between the test results stored it two different files.
Because the report file format is readily parseable by any Haskell program, it could be processed into another format for compatibility with existing tools. _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe