By 'test size reduction' I mean the 'shrink' function. It seems to me (as I said, first Haskell program and no experience with quickcheck) that it works nicely with a topdown generation of the test, but I don't see how to easily generate correct programs that way. Even if you cannot release your tests, maybe you can help me with a very simple case. Consider a trivial AST. A program is a possibly nested block. Each block is a bunch of declarations of variables and some use of them data Block = Block [Declaration] [Statement] data Declaration = Var String String data Statement = Statement Block | Use String How one would generate things like: Block [Var "a" "int"] [Use "a"] -- here a is declared in the same block Block [] [Statement Block [Var "s" "int"] [Statement Block [] [Use "s"]]] -- here s is declared in some other visible scope etc. Or am I trying to approach the problem from the wrong angle? On Mon, Feb 16, 2015 at 12:02 PM, Andrey Chudnov wrote:

I use QuichCheck for compiler testing where I generate random, but well-formed programs and check some high-level syntactic properties on results. The QuickCheck instance is open-source (see language-ecmascript), but the compiler-test code is closed-source at this time. Still, I found that it's not the ultimate answer: many properties are hard to formalize, so I have to resort to unit tests. I'm not sure what you mean by "how this would play with test size reduction". I think it's worth giving a try, but keep in mind that you might still need to use unit tests.

Let me know if you have any questions.

/Andrey

On 02/16/2015 11:53 AM, Maurizio Vitale wrote:

I'm starting to work on my first real haskell program (I've only RWH exercises under my belt) and wondering whether people use quickcheck at all for compiler testing.

I've seen uses of quickcheck for testing parsers, but I'm interested in generating well-formed programs (e.g. random programs with all declarations in reasonable random places). This could be used to test passes other than parsing (or even parsing, for languages that needs to distinguish identifiers, like the 'typedef' problem in C/C++).

The only thing I can think of, is to use quickcheck for randomly generating statements, go over them and figure out free variables (incl. functions) and generate declarations in random places for them. But I'm not sure how this would play with test size reduction and doesn't look like a nice solution anyhow.

Any idea or pointers to examples? or should I give up on quickcheck for this and just do direct testing?

Thanks,

Maurizio

_______________________________________________ Haskell-Cafe mailing listHaskell-Cafe@haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

If you just need an example for generating and shrinking arbitrary ASTs, you can have a look here: https://github.com/jswebtools/language-ecmascript/blob/master/src/Language/E... . Note that I use the testing-feat package to generate Gen instances --- it tends to do better (explore interesting cases earlier) than handwritten code --- and then fix-up incorrect ASTs (another option is to simply discard them). On 02/16/2015 12:33 PM, Maurizio Vitale wrote:

By 'test size reduction' I mean the 'shrink' function. It seems to me (as I said, first Haskell program and no experience with quickcheck) that it works nicely with a topdown generation of the test, but I don't see how to easily generate correct programs that way.

Even if you cannot release your tests, maybe you can help me with a very simple case. Consider a trivial AST. A program is a possibly nested block. Each block is a bunch of declarations of variables and some use of them

data Block = Block [Declaration] [Statement]

data Declaration = Var String String

data Statement = Statement Block | Use String

How one would generate things like:

Block [Var "a" "int"] [Use "a"] -- here a is declared in the same block

Block [] [Statement Block [Var "s" "int"] [Statement Block [] [Use "s"]]] -- here s is declared in some other visible scope

etc.

Or am I trying to approach the problem from the wrong angle?

On Mon, Feb 16, 2015 at 12:02 PM, Andrey Chudnov mailto:achudnov@gmail.com> wrote:

I use QuichCheck for compiler testing where I generate random, but well-formed programs and check some high-level syntactic properties on results. The QuickCheck instance is open-source (see language-ecmascript), but the compiler-test code is closed-source at this time. Still, I found that it's not the ultimate answer: many properties are hard to formalize, so I have to resort to unit tests. I'm not sure what you mean by "how this would play with test size reduction". I think it's worth giving a try, but keep in mind that you might still need to use unit tests.

Let me know if you have any questions.

/Andrey

On 02/16/2015 11:53 AM, Maurizio Vitale wrote:

...

I'm starting to work on my first real haskell program (I've only RWH exercises under my belt) and wondering whether people use quickcheck at all for compiler testing.

I've seen uses of quickcheck for testing parsers, but I'm interested in generating well-formed programs (e.g. random programs with all declarations in reasonable random places). This could be used to test passes other than parsing (or even parsing, for languages that needs to distinguish identifiers, like the 'typedef' problem in C/C++).

The only thing I can think of, is to use quickcheck for randomly generating statements, go over them and figure out free variables (incl. functions) and generate declarations in random places for them. But I'm not sure how this would play with test size reduction and doesn't look like a nice solution anyhow.

Any idea or pointers to examples? or should I give up on quickcheck for this and just do direct testing?

Thanks,

Maurizio

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org mailto:Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org mailto:Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

Do you know if there is an implementation? On 02/17/2015 03:15 AM, Paul Brauner wrote:

There's a recent paper by Jonas Duregård et al where they introduce a variant of feat that allows for efficient discarding of unwanted values, à la lazy Smallcheck. The motivating example of the paper is exactly what you're looking for: discarding ill-formed programs.

On Mon, Feb 16, 2015, 19:44 Andrey Chudnov mailto:achudnov@gmail.com> wrote:

If you just need an example for generating and shrinking arbitrary ASTs, you can have a look here: https://github.com/jswebtools/language-ecmascript/blob/master/src/Language/E... . Note that I use the testing-feat package to generate Gen instances --- it tends to do better (explore interesting cases earlier) than handwritten code --- and then fix-up incorrect ASTs (another option is to simply discard them).

On 02/16/2015 12:33 PM, Maurizio Vitale wrote:

...

By 'test size reduction' I mean the 'shrink' function. It seems to me (as I said, first Haskell program and no experience with quickcheck) that it works nicely with a topdown generation of the test, but I don't see how to easily generate correct programs that way.

Even if you cannot release your tests, maybe you can help me with a very simple case. Consider a trivial AST. A program is a possibly nested block. Each block is a bunch of declarations of variables and some use of them

data Block = Block [Declaration] [Statement]

data Declaration = Var String String

data Statement = Statement Block | Use String

How one would generate things like:

Block [Var "a" "int"] [Use "a"] -- here a is declared in the same block

Block [] [Statement Block [Var "s" "int"] [Statement Block [] [Use "s"]]] -- here s is declared in some other visible scope

etc.

Or am I trying to approach the problem from the wrong angle?

On Mon, Feb 16, 2015 at 12:02 PM, Andrey Chudnov mailto:achudnov@gmail.com> wrote:

I use QuichCheck for compiler testing where I generate random, but well-formed programs and check some high-level syntactic properties on results. The QuickCheck instance is open-source (see language-ecmascript), but the compiler-test code is closed-source at this time. Still, I found that it's not the ultimate answer: many properties are hard to formalize, so I have to resort to unit tests. I'm not sure what you mean by "how this would play with test size reduction". I think it's worth giving a try, but keep in mind that you might still need to use unit tests.

Let me know if you have any questions.

/Andrey

On 02/16/2015 11:53 AM, Maurizio Vitale wrote:

...

I'm starting to work on my first real haskell program (I've only RWH exercises under my belt) and wondering whether people use quickcheck at all for compiler testing.

I've seen uses of quickcheck for testing parsers, but I'm interested in generating well-formed programs (e.g. random programs with all declarations in reasonable random places). This could be used to test passes other than parsing (or even parsing, for languages that needs to distinguish identifiers, like the 'typedef' problem in C/C++).

The only thing I can think of, is to use quickcheck for randomly generating statements, go over them and figure out free variables (incl. functions) and generate declarations in random places for them. But I'm not sure how this would play with test size reduction and doesn't look like a nice solution anyhow.

Any idea or pointers to examples? or should I give up on quickcheck for this and just do direct testing?

Thanks,

Maurizio

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org mailto:Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org mailto:Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org mailto:Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

also a longer thesis (which I believe is the backtracking optimization they refer to in the paper) is at http://publications.lib.chalmers.se/records/fulltext/157525.pdf I'll have to play with a very small language and see where I get. Thanks to everybody, and if there're other examples around I'll be glad to know about them. On Tue, Feb 17, 2015 at 3:15 AM, Paul Brauner wrote:

There's a recent paper by Jonas Duregård et al where they introduce a variant of feat that allows for efficient discarding of unwanted values, à la lazy Smallcheck. The motivating example of the paper is exactly what you're looking for: discarding ill-formed programs.

On Mon, Feb 16, 2015, 19:44 Andrey Chudnov wrote:

...

If you just need an example for generating and shrinking arbitrary ASTs, you can have a look here: https://github.com/jswebtools/language-ecmascript/blob/master/src/Language/E... . Note that I use the testing-feat package to generate Gen instances --- it tends to do better (explore interesting cases earlier) than handwritten code --- and then fix-up incorrect ASTs (another option is to simply discard them).

On 02/16/2015 12:33 PM, Maurizio Vitale wrote:

By 'test size reduction' I mean the 'shrink' function. It seems to me (as I said, first Haskell program and no experience with quickcheck) that it works nicely with a topdown generation of the test, but I don't see how to easily generate correct programs that way.

Even if you cannot release your tests, maybe you can help me with a very simple case. Consider a trivial AST. A program is a possibly nested block. Each block is a bunch of declarations of variables and some use of them

data Block = Block [Declaration] [Statement]

data Declaration = Var String String

data Statement = Statement Block | Use String

How one would generate things like:

Block [Var "a" "int"] [Use "a"] -- here a is declared in the same block

Block [] [Statement Block [Var "s" "int"] [Statement Block [] [Use "s"]]] -- here s is declared in some other visible scope

etc.

Or am I trying to approach the problem from the wrong angle?

On Mon, Feb 16, 2015 at 12:02 PM, Andrey Chudnov wrote:

...

I use QuichCheck for compiler testing where I generate random, but well-formed programs and check some high-level syntactic properties on results. The QuickCheck instance is open-source (see language-ecmascript), but the compiler-test code is closed-source at this time. Still, I found that it's not the ultimate answer: many properties are hard to formalize, so I have to resort to unit tests. I'm not sure what you mean by "how this would play with test size reduction". I think it's worth giving a try, but keep in mind that you might still need to use unit tests.

Let me know if you have any questions.

/Andrey

On 02/16/2015 11:53 AM, Maurizio Vitale wrote:

I'm starting to work on my first real haskell program (I've only RWH exercises under my belt) and wondering whether people use quickcheck at all for compiler testing.

I've seen uses of quickcheck for testing parsers, but I'm interested in generating well-formed programs (e.g. random programs with all declarations in reasonable random places). This could be used to test passes other than parsing (or even parsing, for languages that needs to distinguish identifiers, like the 'typedef' problem in C/C++).

The only thing I can think of, is to use quickcheck for randomly generating statements, go over them and figure out free variables (incl. functions) and generate declarations in random places for them. But I'm not sure how this would play with test size reduction and doesn't look like a nice solution anyhow.

Any idea or pointers to examples? or should I give up on quickcheck for this and just do direct testing?

Thanks,

Maurizio

_______________________________________________ Haskell-Cafe mailing listHaskell-Cafe@haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe