I've been working on getting test running Cabalized and finally have a
patch to send. Extracting tests from the source code is, I think, an
independent problem.
http://jeffrey.yasskin.info/darcs/filepath/Setup.hs has the main
instructions for running the test (with the same test extraction as
the original GenTests.hs), and
http://jeffrey.yasskin.info/darcs/filepath/SingleFileOps.hs has some
operations that I'd like to add to Cabal itself. With some more
experience writing tests like this, we'll definitely want to extract
most of this Setup.hs into a shared module.
The basic idea is that I want to base the tests as much as possible on
production code, but I may need to #define TESTING or some other
symbol in a particular file to get that file's tests to run. So each
file "foo.hs" may be compiled with the 'tag' "foo" (practically, into
the dist/build/foo directory) with -DTESTING, and then its test is run
in the context of ["foo", ""] (look for "foo" object files, and then
the results of `Setup.hs build`). Putting testing builds in a
different place also makes sure that they don't get installed or
released accidentally.
What would be the best way to proceed? Is this system overkill and I
should start over with something simpler (either a single -DTESTING
build tree or just `runhaskell -DTESTING`)? Should I clean up
SingleFileOps and submit it to Cabal? Should I take what I have and
apply it to base?
This patch is based on http://www.cs.york.ac.uk/fp/darcs/filepath/, so
it's a little out of date. Neil, if you're interested in accepting
this for the version of FilePath on darcs.haskell.org, I'll get you a
patch that works there in the next week.
On 3/18/07, Neil Mitchell
Hi
This gives a list of properties, which are automatically extracted from the documentation, and checked using QuickCheck. All the tool is in the repo. If anyone wanted to generalise the code in there, it would be handy!
Neil, your extractor looks cool, but I wonder how it'll work with modules like Data.Set whose (commented out) tests define some Arbitrary instances and some helper functions in addition to the actual properties. My first inclination would have been to define the properties in code, perhaps "#ifdef TESTING"ed out, and try to teach Haddock to pull them into the documentation. But yours is definitely simpler and probably worth pursuing until it obviously stops working.
FilePath also has that issue, I do:
module FilePath( #ifdef TESTING properties and instances for testing #endif
) where
Then I just define TESTING when running the test. This has the advantage that the tests at least always compile, even as I make other changes. In reality, FilePath only requires a couple of things (the drive properties) to be exported for testing but not for real use.
I don't see much complication in extending my approach. In fact, I suspect that my FilePath test extractor is overly complex, since it allows:
Windows: property Posix: property property
For Windows properties it added System.FilePath.Windows.functions, ditto for Posix, and for normal ones it generates the test twice, once with Windows and once with Posix. It also detects if a property has any free variables, and if not changes it to a single assertion, rather than a quickcheck property.
I think it is very handy, and integrating it with Cabal and removing the special cases that FilePath requires is probably well worth the effort! I would certainly like such a tool...
Thanks
Neil
-- Namasté, Jeffrey Yasskin