On Sat, 2008-05-31 at 10:10 -0700, Don Stewart wrote:
You're on your own on this one, Ian. QuickCheck is a cheap, simple way to get good coverage, that's why Haskell library writers use it.
Please include tests, Krasimir!
Ian does have a point. We're talking about two different testing contexts: * GHC nightly builds * tests run on commits by developers I absolutely agree with you Don about the second context. That's the one eg for bytestring where we found lots of our own stupid mistakes before they caused segfaults for end users. It vital for initial development and for making subsequent changes to the library. On the other hand Ian is quite right that for GHC nightly builds (where you're not actually making any changes to the module) it does seem like a lot of wasted effort to run the same tests again and again. Perhaps we can see a way to run the right tests in the right context. If there have been changes then the full QC testsuite is demanded. If on the other hand we're just looking at a system level integration test then perhaps we do not need to run all the tests at a maximum level of coverage. Perhaps we could take a probabilistic approach. How about a QC test driver that runs say 2 instances of each QC property at the same QC Aribtrary 'size' as you'd hit when you do the full 100 tests. That way each nightly run tests a different sample and we don't just keep hitting the 0/[] cases (though we will still hit 0/[] sometimes, the 'size' parameter is a maximum not minimum). It should still catch the obvious integration bugs (like changed semantics of imported modules). Then we can do 'full' runs at important junctures like release candidates. Duncan