Re: [Haskell-cafe] ANNOUNCE: test-framework-golden-1.1

On Sun, Oct 07, 2012 at 09:09:07PM +0300, Roman Cheplyaka wrote:

* Simon Hengel [2012-10-07 15:45:21+0200]

...

On Fri, Oct 05, 2012 at 05:17:18PM +0300, Roman Cheplyaka wrote:

...

I can do that indeed, and I guess I could reimplement everything I have at the moment on top of HUnit.

However, an important part of functionality isn't there at the moment — golden file management. You should be able to say, "for this test, take its current output and write it to the corresponding golden file".

In order to do that, you need to have access to the list of golden tests in the suite. This is where implementation details of different test frameworks start to matter. Probably one can make an abstraction over test frameworks that would give the list of all golden tests. (Although when you start abstracting over test frameworks, which are abstractions themselves, it becomes somewhat funny.)

Ok, makes sense.

I'm looking forward to give it a try, and see how it compares to using operating system primitives (say `cp') for "golden file management".

1. You often want to update not just one test, but all, or some of the tests (when you've made a change and verified that the changes in output are expected). Doing it in command line is certainly possible, but not trivial nor convenient. 2. For some tests (like goldenVsString) the output is not captured in a file, so using cp directly is not possible.

Yes and yes. I have no fixed expectations nor any idea how an "ideal" interface would look like, but I guess there is room for improvement. So I'm really looking forward to try it ;)

...

...

Speaking of such functionality, correct me if I'm wrong, but neither HUnit nor hspec won't be able to support it anyway, because they represent tests as opaque IO actions.

It would be easy to extend Hspec to support this in the same way you extend test-framework to support this. It requires existentials; the only substantial difference that I can see is that test-framework already uses existentials, while Hspec does not.

Well, if you are willing to make this change, then I'll try to do my part of the job and expose a useful abstraction.

If it gives us something that is useful from a users perspective, I'm happy to make that change. I would hope that something like [1] works, e.g.: instance Example GoldenTest where evaluateExample c = evaluateExample c . goldenTestToHUnitAssetion exampleMetadata = Just . Metadata That way the Typeable instance is optional. Personally, I still think that it may be a good idea to first explore the design space with test-framework before trying to abstract over it. Cheers, Simon [1] https://github.com/sol/hspec/commit/6927f642aea44803b57c2b77548931f6865b0c38