On Wed, Jan 5, 2011 at 3:02 PM, Jonathan Geddes wrote:

...

The Haskell type system is simply not rich enough to guarantee everything you might need. Despite all this, I suspect that since Haskell is at a higher level of abstraction than other languages, the tests in Haskell must be at a correspondingly higher level than the tests in other languages. I can see that such tests would give great benefits to the development process. I am convinced that I should try to write such tests. But I still think that Haskell makes a huge class of tests unnecessary.

The way I think about this is that you want to write tests for things that can not be usefully represented in the type system. If you have a parametrically typed function, then the type system is doing a lot of useful "testing" for you. If you want to make sure that you properly parse documents in a given format, then having a bunch of examples that feed into unit tests is a smart move. Anthony

On Wed, Jan 05, 2011 at 10:27:29PM +0100, Gregory Collins wrote:

Once I had written the test harness, I spent literally less than a half-hour setting this up. Highly recommended, even if it is a (blech) Java program. Testing is one of the few areas where I think our "software engineering" tooling is on par with or exceeds that which is available in other languages.

Indeed, I have found this to be true as well, and have been trying to explain it to non-Haskellers. Though I would also rank the memory/space profiler very high compared to what is available for some other languages. And note, it's also easy to integrate with the Python-based buildbot, if one doesn't want to run Java :) regards, iustin

On Wed, Jan 5, 2011 at 6:41 PM, John Zabroski wrote:

5. I have a hard time understanding statements like "The difficulties in unit testing OO code is coaxing objects into the correct state to test a particular property." Difficulty in unit testing OO code is best documented in Robert Binder's tome [1], which is easily the best text on testing I've ever read and never gets cited by bloggers and other Internet programmarazzi (after all, who has time to read 1,500 pages on testing when you have to maintain a blog). Moreover, you should not be mocking objects. That will lead to a combinatorial explosion in tests and likely reveal that your object model leaks encapsulation details (think about it). Mock the role the object plays in the system instead; this is kind of a silly way to say "use abstraction" but I've found most people need to hear a good idea 3 different ways in 3 different contexts before they can apply it beyond one playground trick.

7. Difficulty in testing objects depends on how you describe object behavior, and has nothing to do with any properties of objects as compared with abstract data types! For example, if object actions are governed by an event system, then to test an interaction, you simply mock the event queue manager. This is because you've isolated your test to three variants: (A) the state prior to an atomic action, (B) the state after that action, and (C) any events the action generates. This is really not any more complicated than using QuickCheck, but unfortunately most programmers are only familiar with using xUnit libraries for unit testing and they have subdued the concept of "unit testing" to a common API that is not particularly powerful. Also, note earlier my dislike of the argument that "The difficulties in unit testing OO code is coaxing objects into the correct state to test a particular property."; under this testing methodology, there is no "particular property" to test, since the state of the application is defined in terms of all the object's attributes *after* the action has been processed. It doesn't make much sense to just test one property. It's called unit testing, not property testing.

One small update for pedagogic purposes: Testing properties is really just a form of testing called negative testing; testing that something doesn't do something it shouldn't do. The testing I covered above describes positive testing. Negative testing is always going to be difficult, regardless of how you abstract your system and what language you use. Think about it!

I would supplement this excellent list of advices with an emphasis on the first one: Test-Driven Development is *not* testing, TDD is a *design* process. Like you said, it is a discipline of thought that forces you first to express your intent with a test, second to write the simplest thing that can possibly succeed, third to remove duplication and refactor your code. It happens that this process is somewhat different in Haskell than in say Java, and actually much more fun and interesting thanks to the high signal-to-noise ratio syntax of Haskell (once you get acquainted with it of course) and its excellent support for abstaction, duplication removal, generalization and more generally refactoring (tool support may be better though...). For example, if I were to develop map in TDD (which I did actually...), I could start with the following unit test:

map id [] ~?= []

which I would make pass very simply by copy and pasting, only changing one symbol.

map id [] = []

Then I would add a failing test case:

TestList [ map id [] ~?= [] , map id [1] ~?= [1] ]

which I would make pass with, once again simple copy-pasting:

map id [] = [] map id [1] = [1]

Next test could be :

TestList [ map id [] ~?= [] , map id [1] ~?= [1], , map id [2] ~?= [2] ]

Which of course would pass with:

map id [] = [] map id [1] = [1] map id [2] = [2]

then I would notice an opportunity for refactoring:

map id [] = [] map id [x] = [x]

etc, etc...Sound silly? Sure it is at first sight, and any self-respecting haskeller would write such a piece, just like you said, without feeling the need to write the tests, simply by stating the equations about map. The nice thing with haskell is that it has a few features that helps in making those bigger steps in TDD, whereas less gifted languages and platforms requires a lot more experience and self-confidence to "start running": - writing types delivers some of the design burden off the tests, while keeping their intent (being executable instead of laying in dead trees), - quickcheck and friends help you express a whole class of those unit tests in one invariant expression, while keeping the spirit of TDD as one can use the counter-examples produced to drive the code-writing process. <plug> Some people might be interested in http://www.oqube.net/projects/beyond-tdd/ (a session I co-presented at SPA2009) which was an experiment to try bringing the benefits of TDD with quickcheck in haskell to the java world. </plug> Regards, Arnaud PS: On

On Wed, Jan 5, 2011 at 1:27 PM, Gregory Collins wrote:

On Wed, Jan 5, 2011 at 9:02 PM, Jonathan Geddes wrote:

...

Despite all this, I suspect that since Haskell is at a higher level of abstraction than other languages, the tests in Haskell must be at a correspondingly higher level than the tests in other languages. I can see that such tests would give great benefits to the development process. I am convinced that I should try to write such tests. But I still think that Haskell makes a huge class of tests unnecessary.

I write plenty of tests. Where static typing helps is that of course I don't write tests for type errors, and more things are type errors than might be in other languages (such as incomplete cases). But I write plenty of tests to verify high level relations: with this input, I expect this kind of output. A cheap analogue to "test driven" that I often do is "type driven", I write down the types and functions with the hard bits filled in with 'undefined'. Then I :reload the module until it typechecks. Then I write tests against the hard bits, and run the test in ghci until it passes. However:

QuickCheck especially is great because it automates this tedious work: it fuzzes out the input for you and you get to think in terms of higher-level invariants when testing your code. Since about six months ago with the introduction of JUnit XML support in test-framework, we also have plug-in instrumentation support with continuous integration tools like Hudson:

Incidentally, I've never been able to figure out how to use QuickCheck. Maybe it has more to do with my particular app, but QuickCheck seems to expect simple input data and simple properties that should hold relating the input and output, and in my experience that's almost never true. For instance, I want to ascertain that a function is true for "compatible" signals and false for "incompatible" ones, where the definition of compatible is quirky and complex. I can make quickcheck generate lots of random signals, but to make sure the "compatible" is right means reimplementing the "compatible" function. Or I just pick a few example inputs and expected outputs. To get abstract enough that I'm not simply reimplementing the function under test, I have to move to a higher level, and say that notes that have incompatible signals should be distributed among synthesizers so they don't make each other sound funny. But now it's too high level: I need a definition of "sound funny" and a model of a synthesizer... way too much work and it's fuzzy anyway. And at this level the input data is complex enough that I'd have to spend a lot of time writing and tweaking (and testing!) the data generator to verify it's covering the part of the state space I want to verify. I keep trying to think of ways to use QuickCheck, and keep failing. In my experience, the main work of testing devolves to a library of functions to create the input data, occasionally very complex, and a library of functions to extract the interesting bits from the output data, which is often also very complex. Then it's just a matter of 'equal (extract (function (generate input data))) "abstract representation of output data"'. This is how I do testing in python too, so I don't think it's particularly haskell-specific. I initially tried to use the test-framework stuff and HUnit, but for some reason it was really complicated and confusing to me, so I gave up and wrote my own that just runs all functions starting with 'test_'. It means I don't get to use the fancy tools, but I'm not sure I need them. A standard profile output to go into a tool to draw some nice graphs of performance after each commit would be nice though, surely there is such a thing out there?

On Wed, Jan 5, 2011 at 7:31 PM, Chung-chieh Shan wrote:

Besides those example inputs and expected outputs, what about: If two signals are (in)compatible then after applying some simple transformations to both they remain (in)compatible?  A certain family of signals is always compatible with another family of signals?  Silence is compatible with every signal?  Every non-silent signal is (in)compatible with itself (perhaps after applying a transformation)?

Well, signals are never transformed. Silence is, in fact, not specially compatible. The most I can say is that signals that don't overlap are always compatible. So you're correct in that it's possible to extract properties. However, this particular property, being simple, is also expressed in a simple way directly in the code, so past a couple tests to make sure I didn't reverse any (>)s, I don't feel like it needs the exhaustive testing that quickcheck brings to bear. And basically it's just reimplementing a part of the original function, in this case the first guard... I suppose you could say if I typoed the (>)s in the original definition, maybe I won't in the test version. But this is too low level, what I care about is if the whole thing has the conceptually simple but computationally complex result that I expect. The interesting bug is when the first guard shadows an exception later on, so it turns out it's *not* totally true that non-overlapping signals must be compatible, or maybe my definition of "overlapping" is not sufficiently defined, or defined different ways in different places, or needs to be adjusted, or.... I suppose input fuzzing should be able to flush out things like fuzzy definitions of overlapping I can also say weak things about complex outputs, that they will be returned in sorted order, that they won't overlap, etc. But I those are rarely the interesting complicated things that I really want to test. Even my "signal compatibility" example is relatively amenable to extracting properties, picking some other examples: - Having a certain kind of syntax error will result in a certain kind of error msg, and surrounding expressions will continue to be included in the output. The error msg will include the proper location. So I'd need an Arbitrary to generate the right structure with an error or two and then have code to figure out the reported location from the location in the data structure, and debug all that. There's actually a fair amount of stuff that wants to look for a log msg, like "hit a cache, the only sign of which is a log msg of a certain format". Certainly caches can be tested by asserting that you get the same results with the cache turned off, that's an easy property. - Hitting a certain key sequence results in certain data being entered in the UI. There's nothing particularly "property" like about this, it's too ad-hoc... this seems to apply for all UI-level tests. - There's also a large class of "integration" type tests: I've tested the signal compatibility function separately, but the final proof is that the high-level user input of this shape results in this output, due to do signal compatibility. These are the ones whose failure is the most valuable because they test the emergent behaviour of a set of interacting systems, and that's ultimately the user-visible behaviour and also the place where the results are the most subtle. But those are also the ones that have huge state spaces and, similar to the UI tests, basically ad-hoc relationships between input and output. - Testing for laziness of course doesn't work either. Or timed things. As far as performance goes, some can be tested with tests ("taking the first output doesn't force the entire input" or "a new key cancels the old threads and starts new ones") but some must be tested with profiling and eyeballing the results. QuickCheck seems to fit well when you have small input and output spaces, but complicated stuff in the middle, but still simple relations between the input and output. I think that's why data structures are so easy to QuickCheck. I suppose I should look around for more use of QuickCheck for non-data structures... the examples I've seen have been trivial stuff like 'reverse . reverse = id'.

Seeing all the good discussion on this thread, I think we are missing a TDD page on our Haskell.org wiki. =) Cheers, -- Felipe.

Florian Weimer wrote:

* Jonathan Geddes:

...

When I write Haskell code, I write functions (and monadic actions) that are either a) so trivial that writing any kind of unit/property test seems silly, or are b) composed of other trivial functions using equally-trivial combinators.

You can write in this style in any language which has good support for functional composition (which means some sort of garbage collection and perhaps closures, but strong support for higher-order functions is probably not so important). But this doesn't mean that you don't have bugs. There are a few error patterns I've seen when following this style (albeit not in Haskell):

As you mention, the bugs below can all be avoided in Haskell by using the type system and the right abstractions and combinators. You can't put everything in the type system - at some point, you do have to write actual code - but you can isolate potential bugs to the point that their correctness becomes obvious.

While traversing a data structure (or parsing some input), you fail to make progress and end up in an infinite loop.

Remedy: favor higher-order combinators like fold and map over primitive recursion.

You confuse right and left (or swap two parameters of the same type), leading to wrong results.

Remedy: use more descriptive types, for instance by putting Int into an appropriate newtype. Use infix notation source `link` target.

All the usual stuff about border conditions still applies.

Partial remedy: choose natural boundary conditions, for example and [] = True, or [] = False. But I would agree that this is one of the main use cases for QuickCheck.

Input and output is more often untyped than typed. Typos in magic string constants (such as SQL statements) happen frequently.

Remedy: write magic string only once. Put them in a type-safe combinator library.

Therefore, I think that you cannot really avoid extensive testing for a large class of programming tasks.

Hopefully, the class is not so large anymore. ;)

...

I vehemently disagreed, stating that invariants embedded in the type system are stronger than any other form of assuring correctness I know of.

But there are very interesting invariants you cannot easily express in the type system, such as "this list is of finite length".

Sure, you can. data FiniteList a = Nil | Cons a !(FiniteList a) I never needed to know whether a list is finite, though. It is more interesting to know whether a list is infinite. data InfiniteList a = a :> InfiniteList a

It also seems to me that most Haskell programmers do not bother to turn the typechecker into some sort of proof checker. (Just pick a few standard data structures on hackage and see if they perform such encoding. 8-)

I at least regularly encode properties in the types, even if it's only a type synonym. I also try to avoid classes of bugs by "making them obvious", i.e. organizing my code in such a way that correctness becomes obvious. Regards, Heinrich Apfelmus -- http://apfelmus.nfshost.com