On Wed, Jan 5, 2011 at 7:31 PM, Chung-chieh Shan
Besides those example inputs and expected outputs, what about: If two signals are (in)compatible then after applying some simple transformations to both they remain (in)compatible? A certain family of signals is always compatible with another family of signals? Silence is compatible with every signal? Every non-silent signal is (in)compatible with itself (perhaps after applying a transformation)?
Well, signals are never transformed. Silence is, in fact, not specially compatible. The most I can say is that signals that don't overlap are always compatible. So you're correct in that it's possible to extract properties. However, this particular property, being simple, is also expressed in a simple way directly in the code, so past a couple tests to make sure I didn't reverse any (>)s, I don't feel like it needs the exhaustive testing that quickcheck brings to bear. And basically it's just reimplementing a part of the original function, in this case the first guard... I suppose you could say if I typoed the (>)s in the original definition, maybe I won't in the test version. But this is too low level, what I care about is if the whole thing has the conceptually simple but computationally complex result that I expect. The interesting bug is when the first guard shadows an exception later on, so it turns out it's *not* totally true that non-overlapping signals must be compatible, or maybe my definition of "overlapping" is not sufficiently defined, or defined different ways in different places, or needs to be adjusted, or.... I suppose input fuzzing should be able to flush out things like fuzzy definitions of overlapping I can also say weak things about complex outputs, that they will be returned in sorted order, that they won't overlap, etc. But I those are rarely the interesting complicated things that I really want to test. Even my "signal compatibility" example is relatively amenable to extracting properties, picking some other examples: - Having a certain kind of syntax error will result in a certain kind of error msg, and surrounding expressions will continue to be included in the output. The error msg will include the proper location. So I'd need an Arbitrary to generate the right structure with an error or two and then have code to figure out the reported location from the location in the data structure, and debug all that. There's actually a fair amount of stuff that wants to look for a log msg, like "hit a cache, the only sign of which is a log msg of a certain format". Certainly caches can be tested by asserting that you get the same results with the cache turned off, that's an easy property. - Hitting a certain key sequence results in certain data being entered in the UI. There's nothing particularly "property" like about this, it's too ad-hoc... this seems to apply for all UI-level tests. - There's also a large class of "integration" type tests: I've tested the signal compatibility function separately, but the final proof is that the high-level user input of this shape results in this output, due to do signal compatibility. These are the ones whose failure is the most valuable because they test the emergent behaviour of a set of interacting systems, and that's ultimately the user-visible behaviour and also the place where the results are the most subtle. But those are also the ones that have huge state spaces and, similar to the UI tests, basically ad-hoc relationships between input and output. - Testing for laziness of course doesn't work either. Or timed things. As far as performance goes, some can be tested with tests ("taking the first output doesn't force the entire input" or "a new key cancels the old threads and starts new ones") but some must be tested with profiling and eyeballing the results. QuickCheck seems to fit well when you have small input and output spaces, but complicated stuff in the middle, but still simple relations between the input and output. I think that's why data structures are so easy to QuickCheck. I suppose I should look around for more use of QuickCheck for non-data structures... the examples I've seen have been trivial stuff like 'reverse . reverse = id'.