I agree that this is a huge pain. I've been working on an alternative numeric hierarchy, and as part of that I've wanted a completely automated test suite. What I've done is use template haskell to write boilerplate code for me. Currently, I am verifying around 1000 quickcheck properties, all of which are automatically generated. I suspect that by the time I finish the final number of properties will be between 10000-100000, so full automation for this is a must! Here's a file-by-file breakdown of how it works: ---- https://github.com/mikeizbicki/subhask/blob/master/test/TestSuite.hs This file defines the actual tests. You specify a type to test, and then the template haskell figures out all of the classes it's an instance of and runs the appropriate tests on that type. Eventually, I'd like to automate this a bit more so that you don't have to manually specify types to test, and the template haskell will determine that as well. ---- https://github.com/mikeizbicki/subhask/blob/master/src/SubHask/Algebra.hs Here's the code for the classes. Notice that for each class I've created functions beginning with law_, defn_ or theorem_ that define the required properties of the class. These are the tests that will get automatically run on an instance of that class to verify it obeys the laws. The prefix differences don't mean anything special to my generating code yet. Eventually, I'd like to create a "fast" test suite for quick development and a "thorough" test suite to run before releases, in which case there would be a difference between these prefixes. ---- https://github.com/mikeizbicki/subhask/blob/master/src/SubHask/TemplateHaske... Then I have a bunch of template haskell code to generate test cases automatically from the law_/defn_/theorem_ functions. The `testmap` variable contains all the tests that apply to each class. In theory, this can be generated automatically, however, template haskell isn't yet powerful enough to do this (https://ghc.haskell.org/trac/ghc/ticket/9699). On Wed, Feb 18, 2015 at 12:41 PM, Stuart Hungerford wrote:

Hi,

I'd like to use QuickCheck to test polymorphic properties and I'm wondering if there's any community consenus on how to do this while avoiding creating large amounts of boilerplate test code.

As an example, suppose I have this simple simplistic Monoid typeclass:

class Monoid a where zero :: a (|+|) :: a -> a -> a

And suppose I make these types Monoid instances:

instance Monoid Integer where zero = 0 (|+|) = (+)

instance Monoid Rational where zero = 0 (|+|) = (+)

instance (Monoid a, Monoid b) => Monoid (a, b) where zero = (zero, zero) (|+|) (x, y) (u, v) = (x |+| u, y |+| v)

Using QuickCheck and the Tasty test framework I'd like to test the Monoid laws across the relevant types (I'm only showing one part of one law here) as QuickCheck properties:

prop_left_add_id :: (Eq a, Monoid a) => a -> Bool prop_left_add_id x = zero |+| x == x

monoid_suite :: TestTree monoid_suite = testGroup "monoid" [ testProperty "left additive identity (Integer)" (prop_left_add_id :: Integer -> Bool),

testProperty "left additive identity (Rational)" (prop_left_add_id :: Rational -> Bool),

testProperty "left additive identity ((Integer, Integer))" (prop_left_add_id :: (Integer, Integer) -> Bool), -- ...]

And this is where I start creating a lot of essentially boilerplate code. I'm wondering if there's any consensus on the way to go about this kind of testing?

Researching this issue I've found ways of combining various GHC extensions and the type system to (I think) write the properties more at the type level. There are QuickCheck and Tasty modules that aim to find all properties in a module and test them, but polymorphic properties are automatically defaulted to Integer arguments. I'm also tempted by Template Haskell to write what's essentially code generation templates.

Is there a better way to go about this?

Thanks,

Stu _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

Hi Stuart, If you're testing stand-alone functions, defaulting all type variables to Integer can probably find a counterexample, provided it's not something that depends on properties of an instance of Integer like: prop_assoc a b c = (a + b) + c == a + (b + c) -- fails for Double But another type might find the counterexample faster. See this paper (called Testing Polymorphic Properties): http://publications.lib.chalmers.se/records/fulltext/local_99387.pdf Unfortunately, I don't know of an implementation of the method they describe. Regards, Adam On Wed, Feb 18, 2015 at 3:41 PM, Stuart Hungerford wrote:

Hi,

I'd like to use QuickCheck to test polymorphic properties and I'm wondering if there's any community consenus on how to do this while avoiding creating large amounts of boilerplate test code.

As an example, suppose I have this simple simplistic Monoid typeclass:

class Monoid a where zero :: a (|+|) :: a -> a -> a

And suppose I make these types Monoid instances:

instance Monoid Integer where zero = 0 (|+|) = (+)

instance Monoid Rational where zero = 0 (|+|) = (+)

instance (Monoid a, Monoid b) => Monoid (a, b) where zero = (zero, zero) (|+|) (x, y) (u, v) = (x |+| u, y |+| v)

Using QuickCheck and the Tasty test framework I'd like to test the Monoid laws across the relevant types (I'm only showing one part of one law here) as QuickCheck properties:

prop_left_add_id :: (Eq a, Monoid a) => a -> Bool prop_left_add_id x = zero |+| x == x

monoid_suite :: TestTree monoid_suite = testGroup "monoid" [ testProperty "left additive identity (Integer)" (prop_left_add_id :: Integer -> Bool),

testProperty "left additive identity (Rational)" (prop_left_add_id :: Rational -> Bool),

testProperty "left additive identity ((Integer, Integer))" (prop_left_add_id :: (Integer, Integer) -> Bool), -- ...]

And this is where I start creating a lot of essentially boilerplate code. I'm wondering if there's any consensus on the way to go about this kind of testing?

Researching this issue I've found ways of combining various GHC extensions and the type system to (I think) write the properties more at the type level. There are QuickCheck and Tasty modules that aim to find all properties in a module and test them, but polymorphic properties are automatically defaulted to Integer arguments. I'm also tempted by Template Haskell to write what's essentially code generation templates.

Is there a better way to go about this?

Thanks,

Stu _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe