Re: Automated tests for the libraries?

Hi Jeffrey, Sorry for the delay in replying - I've been busy with other stuff.

I've been working on getting test running Cabalized and finally have a patch to send. Extracting tests from the source code is, I think, an independent problem.

The code for FilePath is going to be including with GHC 6.6.1 which comes out very shortly, so I'd like to put off adding things to Cabal to run the tests until after that. Every change at this time is a little dangerous so I want to keep them minimal. However, after GHC 6.6.1 comes out, and we have a bit more breathing space, I'd love to have the Setup.hs tests run the tests, rather than having a separate .bat file do it. Ideally I want to be consistent with the other packages, so maybe Ian can advise as to how testing should look like? What does testing need to do to get into the GHC buildbot test scripts?

http://jeffrey.yasskin.info/darcs/filepath/SingleFileOps.hs has some operations that I'd like to add to Cabal itself. With some more experience writing tests like this, we'll definitely want to extract most of this Setup.hs into a shared module.

You'll want to look at the Cabal lists for what of the SingleFileOps can go into there.

The basic idea is that I want to base the tests as much as possible on production code, but I may need to #define TESTING or some other symbol in a particular file to get that file's tests to run. So each file "foo.hs" may be compiled with the 'tag' "foo" (practically, into the dist/build/foo directory) with -DTESTING, and then its test is run in the context of ["foo", ""] (look for "foo" object files, and then the results of `Setup.hs build`). Putting testing builds in a different place also makes sure that they don't get installed or released accidentally.

All sounds sensible.

What would be the best way to proceed? Is this system overkill and I should start over with something simpler (either a single -DTESTING build tree or just `runhaskell -DTESTING`)? Should I clean up SingleFileOps and submit it to Cabal? Should I take what I have and apply it to base?

You'd have to ask the Cabal people how much test stuff they want in Cabal, and perhaps have a separate tool that does test extraction if they'd rather it was kept minimal. Thanks Neil

On 3/18/07, Neil Mitchell wrote:

...

Hi

...

...

This gives a list of properties, which are automatically extracted from the documentation, and checked using QuickCheck. All the tool is in the repo. If anyone wanted to generalise the code in there, it would be handy!

Neil, your extractor looks cool, but I wonder how it'll work with modules like Data.Set whose (commented out) tests define some Arbitrary instances and some helper functions in addition to the actual properties. My first inclination would have been to define the properties in code, perhaps "#ifdef TESTING"ed out, and try to teach Haddock to pull them into the documentation. But yours is definitely simpler and probably worth pursuing until it obviously stops working.

FilePath also has that issue, I do:

module FilePath( #ifdef TESTING properties and instances for testing #endif

) where

Then I just define TESTING when running the test. This has the advantage that the tests at least always compile, even as I make other changes. In reality, FilePath only requires a couple of things (the drive properties) to be exported for testing but not for real use.

I don't see much complication in extending my approach. In fact, I suspect that my FilePath test extractor is overly complex, since it allows:

Windows: property Posix: property property

For Windows properties it added System.FilePath.Windows.functions, ditto for Posix, and for normal ones it generates the test twice, once with Windows and once with Posix. It also detects if a property has any free variables, and if not changes it to a single assertion, rather than a quickcheck property.

I think it is very handy, and integrating it with Cabal and removing the special cases that FilePath requires is probably well worth the effort! I would certainly like such a tool...

Thanks

Neil

-- Namasté, Jeffrey Yasskin