* Gianfranco Alongi [2012-10-05 06:57:41+0200]

I'm a big fan of TDD and tend to approach all languages by learning how to TDD in them.

As such, you mention that this is similar, could you please send some links about this?

Sorry, I'm not sure I understand the question. Links about what? The library documentation is available on Hackage (the first link below). If you're asking about TDD in Haskell in general, I doubt I can tell you anything that Google can't. Roman

On Thu, Oct 4, 2012 at 11:55 PM, Roman Cheplyaka wrote:

...

I am glad to announce the first public release of test-framework-golden — a golden testing library.

Hackage: http://hackage.haskell.org/package/test-framework-golden GitHub: https://github.com/feuerbach/test-framework-golden

Golden tests are similar to unit tests (as implemented in HUnit), but the idea is to store the expected result in files (called «golden» files).

I was introduced to the idea of golden testing by Bohdan Vlasyuk at ZuriHac in 2010. Since then I've discovered that this is a nice approach and it is already used in variety of projects (e.g. ghc, haddock).

Surprisingly, not much is written about golden testing, and I've been unable to find any golden testing libraries for Haskell or any other programming language. Every project has its own ad-hoc implementation in Haskell/Python/Shell/etc.

The closest match on the market is Simon Michael's shelltestrunner. But to use it you have to expose the tested functionality via command line, which may be inconvenient.

So this is my attempt at a general golden testing library.

It consists of two modules. Test.Golden has a simple interface that helps you get started very quickly. Test.Golden.Advanced provides a very generic testing function that you can use to implement the testing system you dream about.

The library is integrated with test-framework, so you can use golden tests in addition to SmallCheck/QuickCheck/HUnit tests.

In future there's a plan to support some golden test management capabilities.

Roman

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

* Simon Hengel [2012-10-05 09:07:19+0200]

Hi,

...

I am glad to announce the first public release of test-framework-golden — a golden testing library.

Nice!

...

The library is integrated with test-framework, so you can use golden tests in addition to SmallCheck/QuickCheck/HUnit tests.

I would suggest to rename the modules to

Test.Framework.Golden

and

Test.Framework.Golden.Advanced

to more clearly represent that fact.

According to the test-framework convention, they ought to be named Test.Framework.Providers.Golden and Test.Framework.Providers.Golden.Advanced respectively, and you can see that it was the case in the early versions: http://hackage.haskell.org/package/test-framework-golden-1.0 But it's just too much to type. My justification (which you may or may not buy) is that, unlike, say, Test.Framework.Providers.HUnit, this is not an adaptation of an existing testing library to test-framework, but is a new library that just happens to use test-framework. So it's more like Test.HUnit, although it already subsumes what would become Test.Framework.Providers.Golden. Roman

Cool, looking forward to reading it! Well, the post is already finished: http://ics.p.lodz.pl/~stolarek/blog/2012/10/code-testing-in-haskell/ I was just going to publish it and then your email came up on the list.

I hope you won't forget to cover SmallCheck in your article as well. Being also the maintainer of SmallCheck, I want it to steal some fame from QuickCheck :) Sorry to disappoint you, but I did not mention SmallCheck. So far I'm relying on QuickCheck and didn't feel like I need to look for other testing library. I might be wrong of course.

In theory I totally agree, but:

1. It's hard to guess at the moment how a good interface to the "pure    golden" part should look like. 2. I'd rather avoid doing work when there's no demand. I guess these are valid points.

Jan

I can do that indeed, and I guess I could reimplement everything I have at the moment on top of HUnit. However, an important part of functionality isn't there at the moment — golden file management. You should be able to say, "for this test, take its current output and write it to the corresponding golden file". In order to do that, you need to have access to the list of golden tests in the suite. This is where implementation details of different test frameworks start to matter. Probably one can make an abstraction over test frameworks that would give the list of all golden tests. (Although when you start abstracting over test frameworks, which are abstractions themselves, it becomes somewhat funny.) Speaking of such functionality, correct me if I'm wrong, but neither HUnit nor hspec won't be able to support it anyway, because they represent tests as opaque IO actions. Nor can HTF provide such support — although its TestSort type could be extended to indicate that the test is a golden test, still there's no way to get hold of the golden file name. test-framework will support this once the Typeable constraint is added for tests. Roman * Simon Hengel [2012-10-05 13:43:57+0200]

...

1. It's hard to guess at the moment how a good interface to the "pure golden" part should look like.

Maybe just produce HUnit assertions, e.g.:

goldenVsFile :: FilePath -> FilePath -> IO () -> Assertion

That way it works with plain HUnit

main = runTestTT $ TestLabel "someAction produce desired output $ goldenVsFile "ref.txt" "out.txt" someAction

test-framework

main = defaultMain [ testGroup "someAction" [ testCase "produces some desired output" $ goldenVsFile "ref.txt" "out.txt" someAction ] ]

Hspec

main = hspec $ do describe "someAction" $ do it "produces some desired output" $ do goldenVsFile "ref.txt" "out.txt" someAction

and probably every other current or future Haskell test framework.

Cheers, Simon

* Simon Hengel [2012-10-07 15:45:21+0200]

On Fri, Oct 05, 2012 at 05:17:18PM +0300, Roman Cheplyaka wrote:

...

I can do that indeed, and I guess I could reimplement everything I have at the moment on top of HUnit.

However, an important part of functionality isn't there at the moment — golden file management. You should be able to say, "for this test, take its current output and write it to the corresponding golden file".

In order to do that, you need to have access to the list of golden tests in the suite. This is where implementation details of different test frameworks start to matter. Probably one can make an abstraction over test frameworks that would give the list of all golden tests. (Although when you start abstracting over test frameworks, which are abstractions themselves, it becomes somewhat funny.)

Ok, makes sense.

I'm looking forward to give it a try, and see how it compares to using operating system primitives (say `cp') for "golden file management".

1. You often want to update not just one test, but all, or some of the tests (when you've made a change and verified that the changes in output are expected). Doing it in command line is certainly possible, but not trivial nor convenient. 2. For some tests (like goldenVsString) the output is not captured in a file, so using cp directly is not possible.

...

Speaking of such functionality, correct me if I'm wrong, but neither HUnit nor hspec won't be able to support it anyway, because they represent tests as opaque IO actions.

It would be easy to extend Hspec to support this in the same way you extend test-framework to support this. It requires existentials; the only substantial difference that I can see is that test-framework already uses existentials, while Hspec does not.

Well, if you are willing to make this change, then I'll try to do my part of the job and expose a useful abstraction. Roman