QuickCheck - Extracting data values from tests
Hello all, When using Quickcheck, is there some way to extract generated data values to the IO Monad? I know I can collect and print information about test cases, but that's not enough. Data may be pretty complex, and there may be no parsers for it. If a test suddenly goes wrong, just having it displayed doesn't seem that useful. I'd expect quickCheck to have type: quickCheck :: forall a. (Testable a) => a -> IO [a] Show I could just get the offending data with: please_be_empty <- quickCheck prop_foo Also, even when I'm implementing a generator, I want to see how it is working. Running a verboseCheck on some dummy property helps, but I may want to analyse the data, or some parts of it better - for instance, for many data structures I have alternative "show" functions that take parameters as arguments. Thanks in advance, J.A.
Jorge Adriano Aires
Hello all, When using Quickcheck, is there some way to extract generated data values to the IO Monad?
I know I can collect and print information about test cases, but that's not enough. Data may be pretty complex, and there may be no parsers for it. If a test suddenly goes wrong, just having it displayed doesn't seem that useful.
You may be interested in a QuickCheck hack of mine that saves the offending data value to use immediately in the next test run. You can get the current version with "darcs get http://thunderbird.scannedinavian.com/repos/quickcheck" I've only used this for my own code, so I'd be interested in any feedback. In some cases it's a lot easier to generate a value from a seed and size rather than saving the value in some way that you can restore (ie functions). I've been investigating doing test-driven-development with QuickCheck, saving failing test cases is one step towards that goal. If you have more ideas on that topic, I'd like to hear about it.
Also, even when I'm implementing a generator, I want to see how it is working. Running a verboseCheck on some dummy property helps, but I may want to analyse the data, or some parts of it better - for instance, for many data structures I have alternative "show" functions that take parameters as arguments.
This isn't clear to me, can you give other examples? -- Shae Matijs Erisson - Programmer - http://www.ScannedInAvian.org/ "I will, as we say in rock 'n' roll, run until the wheels come off, because I love what I do." -- David Crosby
Hello all, When using Quickcheck, is there some way to extract generated data values to the IO Monad?
I know I can collect and print information about test cases, but that's not enough. Data may be pretty complex, and there may be no parsers for it. If a test suddenly goes wrong, just having it displayed doesn't seem that useful.
You may be interested in a QuickCheck hack of mine that saves the offending data value to use immediately in the next test run.
Nice! It's different from what I was looking for but also quite usefull.
You can get the current version with "darcs get http://thunderbird.scannedinavian.com/repos/quickcheck" I've only used this for my own code, so I'd be interested in any feedback.
Ok.
In some cases it's a lot easier to generate a value from a seed and size rather than saving the value in some way that you can restore (ie functions).
I've been investigating doing test-driven-development with QuickCheck, saving failing test cases is one step towards that goal. If you have more ideas on that topic, I'd like to hear about it.
Well, returning (part of) the generated data is one of them :)
Also, even when I'm implementing a generator, I want to see how it is working. Running a verboseCheck on some dummy property helps, but I may want to analyse the data, or some parts of it better - for instance, for many data structures I have alternative "show" functions that take parameters as arguments.
This isn't clear to me, can you give other examples?
Not sure which part is not clear... I'll just try to explain each of them. Lets say I'm implementing a generators for Graphs.
Also, even when I'm implementing a generator, I want to see how it is working. I want to check if the generated Graphs are like I intended them to be.
Running a verboseCheck on some dummy property helps, but I may verboseCheck by default prints all the data. I can run it on a dummy function that always returns True to see what kind of data I'm getting.
want to analyse the data, or some parts of it better May want to print the 'actual graphs' on the screen (ASCII art, or maybe using some function that calls Gnuplot). Then I may want to check in more detail the info in contained in some of the nodes. Then I may decide to run some functions on it.
many data structures I have alternative "show" functions that take parameters as arguments. Like I just said, I may want to "show" the graph in many ways.
But there are more possibilities. Why limitate the usefulness of QuickCheck? Suppose I just implemented generators for a few kinds of terms and formulas to test some properties. Now I want to benchmark a couple of different unification functions... I'd expect to be able to use my generator for that. Unless I'm missing something, I cannot. Am I right? J.A.
On Fri, 3 Sep 2004 01:35:45 +0100, Jorge Adriano Aires
Not sure which part is not clear... I'll just try to explain each of them. Lets say I'm implementing a generators for Graphs.
Also, even when I'm implementing a generator, I want to see how it is working. I want to check if the generated Graphs are like I intended them to be.
Running a verboseCheck on some dummy property helps, but I may verboseCheck by default prints all the data. I can run it on a dummy function that always returns True to see what kind of data I'm getting.
want to analyse the data, or some parts of it better May want to print the 'actual graphs' on the screen (ASCII art, or maybe using some function that calls Gnuplot). Then I may want to check in more detail the info in contained in some of the nodes. Then I may decide to run some functions on it.
many data structures I have alternative "show" functions that take parameters as arguments. Like I just said, I may want to "show" the graph in many ways.
But there are more possibilities. Why limitate the usefulness of QuickCheck? Suppose I just implemented generators for a few kinds of terms and formulas to test some properties. Now I want to benchmark a couple of different unification functions... I'd expect to be able to use my generator for that. Unless I'm missing something, I cannot. Am I right?
The generate function is exported: generate :: Int -> StdGen -> Gen a -> a Regards, Martin
participants (3)
-
Jorge Adriano Aires -
Martin Sjögren -
Shae Matijs Erisson