Re: [Haskell-cafe] What is the state of the art in testing code generation?

The first impediment to using QuickCheck is that my EDSL is typed. I don't know how to generate random terms in the presence of types. The second impediment is that, although I can indeed test trivial properties, I don't know how to test functionality to its full level of sophistication. How can I test a left join, for example, without reimplementing left join functionality within Haskell? (Indeed I may end up doing this, but would prefer to find an easier path ...) Tom On Mon, Jul 14, 2014 at 11:26:14AM +0200, Vo Minh Thu wrote:

I think there is still some opportunity for something like QuickCheck in your case. Certainly you can use QC to generate expressions/statements in your EDSL. Then if you can also generate schemas and data you should be able to write down some properties, for instance that some class of queries on empty tables should return no rows.

A special case of this, and a more specific example, is very similar to the introductory examples to QC: it must be possible to retrieve a row after inserting it in an empty table, or deleting it after inserting it must leave the table unchanged.

Even if you don't go as far as writing properties that involves the schema/data generation, generating arbitrary valid AST that must compile successfully to SQL is interesting.

2014-07-11 19:58 GMT+02:00 Tom Ellis :

...

I am implementing an EDSL that compiles to SQL and I am wondering what is the state of the art in testing code generation.

All the Haskell libraries I could find that deal with SQL generation are tested by implementing multiple one-off adhoc queries and checking that when either compiled to SQL or run against a database they give the expected, prespecified result.

* https://github.com/prowdsponsor/esqueleto/blob/master/test/Test.hs * https://github.com/m4dc4p/haskelldb/blob/master/test/TestCases.hs * https://github.com/yesodweb/persistent/blob/master/persistent-test/SumTypeTe...

I couldn't find any tests for groundhog.

* https://github.com/lykahb/groundhog

I also had a look at Javascript generators. They take a similar adhoc, one-off approach.

* https://github.com/valderman/haste-compiler/tree/master/Tests * https://github.com/faylang/fay/tree/master/tests

Is this the best we can do in Haskell? Certainly it seems hard to use a QuickCheck/SmallCheck approach for this purpose. Is there any way this kind of testing can be automated or made more robust?