Automated tests for the libraries?

newer
anyone interested in developing a...

Jeffrey Yasskin

16 Mar 2007 16 Mar '07

5:39 a.m.

Looking through the libraries, I see a lot of commented out imports of QuickCheck and prop_foo definitions, but no way to run them all without a lot of manual editing of files. There are also a few tests in testsuite/tests/libraries/Data/, but they don't even build with ghc-6.6 because of an import of the hidden module Distribution.Compat.FilePath. It seems like this would make it hard to ensure that things work on more than a few platforms. Is there some automated test suite for the libraries that I'm just missing? Is there a plan for developing one? If not, I'll start a wiki page for ideas and try to get a rudimentary one put together. Jeffrey

Show replies by date

dons＠cse.unsw.edu.au

16 Mar 16 Mar

5:48 a.m.

jyasskin:

...

Looking through the libraries, I see a lot of commented out imports of QuickCheck and prop_foo definitions, but no way to run them all without a lot of manual editing of files. There are also a few tests in testsuite/tests/libraries/Data/, but they don't even build with ghc-6.6 because of an import of the hidden module Distribution.Compat.FilePath. It seems like this would make it hard to ensure that things work on more than a few platforms.

Is there some automated test suite for the libraries that I'm just missing? Is there a plan for developing one? If not, I'll start a wiki page for ideas and try to get a rudimentary one put together.

A number of libraries are tested in the 'testsuite' that comes with ghc. The plan is to move these back into the relevant library repositories though. Ian, what's the plan on this? -- Don

Ian Lynagh

2:24 p.m.

On Fri, Mar 16, 2007 at 04:48:41PM +1100, Donald Bruce Stewart wrote:

...

jyasskin:

...
in testsuite/tests/libraries/Data/, but they don't even build with ghc-6.6 because of an import of the hidden module Distribution.Compat.FilePath.

This is a bug in the Cabal package.conf.in, which lists it as hidden rather than exposed. I expect we'll fix this for 6.6.1.

...

...
Is there some automated test suite for the libraries that I'm just missing? Is there a plan for developing one? If not, I'll start a wiki page for ideas and try to get a rudimentary one put together.

A number of libraries are tested in the 'testsuite' that comes with ghc. The plan is to move these back into the relevant library repositories though.

Ian, what's the plan on this?

SimonM seems to have just been doing this a directory at a time, putting the tests in libraries/foo/tests by the looks of it. I don't think there are any major difficulties with finishing it, it just needs someone to spend some time doing it. Thanks Ian

Paul Johnson

7:35 p.m.

...

Is there some automated test suite for the libraries that I'm just missing? Is there a plan for developing one? If not, I'll start a wiki page for ideas and try to get a rudimentary one put together. I think that QuickCheck properties, or at least an abbreviated version of them, should be in the documentation. See my RangedSet library (http://ranged-sets.sourceforge.net/) for an example of the kind of

Jeffrey Yasskin wrote: thing I mean. However I have to maintain the correspondence between the actual properties and the Haddock comments manually, which is a pain. I have in mind an automatic tool along the lines of Haddock, but able to extract QuickCheck properties from comments. Ideally it would also be able to fill in default types so that [a] in a type signature becomes [Int] in the code without having to appear in the Haddock-generated docs. I looked into building something like this once, but the starting point would have had to be Haddock, and I didn't fancy getting into its internals. Paul.

Neil Mitchell

7:38 p.m.

Hi Paul

...

I think that QuickCheck properties, or at least an abbreviated version of them, should be in the documentation. See my RangedSet library (http://ranged-sets.sourceforge.net/) for an example of the kind of thing I mean. However I have to maintain the correspondence between the actual properties and the Haddock comments manually, which is a pain.

See my FilePath library: http://www-users.cs.york.ac.uk/~ndm/projects/libraries.php For example: http://www-users.cs.york.ac.uk/~ndm/projects/filepath/System-FilePath-Versio... This gives a list of properties, which are automatically extracted from the documentation, and checked using QuickCheck. All the tool is in the repo. If anyone wanted to generalise the code in there, it would be handy! Thanks Neil

Jeffrey Yasskin

18 Mar 18 Mar

1:54 a.m.

On 3/16/07, Neil Mitchell wrote:

...

Hi Paul

...
I think that QuickCheck properties, or at least an abbreviated version of them, should be in the documentation. See my RangedSet library (http://ranged-sets.sourceforge.net/) for an example of the kind of thing I mean. However I have to maintain the correspondence between the actual properties and the Haddock comments manually, which is a pain.

See my FilePath library: http://www-users.cs.york.ac.uk/~ndm/projects/libraries.php

For example: http://www-users.cs.york.ac.uk/~ndm/projects/filepath/System-FilePath-Versio...

This gives a list of properties, which are automatically extracted from the documentation, and checked using QuickCheck. All the tool is in the repo. If anyone wanted to generalise the code in there, it would be handy!

Thanks

Neat! I've started http://haskell.org/haskellwiki/Library_tests linked from Library_submissions to record this stuff. Feel free to rearrange that of course. The biggest problem I see with the tests that SimonM is moving is that they have to be run inside a built GHC source tree, which makes it hard to work just on the libraries or test them on another compiler. Neil, your extractor looks cool, but I wonder how it'll work with modules like Data.Set whose (commented out) tests define some Arbitrary instances and some helper functions in addition to the actual properties. My first inclination would have been to define the properties in code, perhaps "#ifdef TESTING"ed out, and try to teach Haddock to pull them into the documentation. But yours is definitely simpler and probably worth pursuing until it obviously stops working. It would be nice to use Cabal to run these tests since it already knows how to pick a compiler. I'll start on a runTests hook using GenTests.hs. -- Namasté, Jeffrey Yasskin

Neil Mitchell

5:15 p.m.

...

...
This gives a list of properties, which are automatically extracted from the documentation, and checked using QuickCheck. All the tool is in the repo. If anyone wanted to generalise the code in there, it would be handy!

Neil, your extractor looks cool, but I wonder how it'll work with modules like Data.Set whose (commented out) tests define some Arbitrary instances and some helper functions in addition to the actual properties. My first inclination would have been to define the properties in code, perhaps "#ifdef TESTING"ed out, and try to teach Haddock to pull them into the documentation. But yours is definitely simpler and probably worth pursuing until it obviously stops working.

FilePath also has that issue, I do: module FilePath( #ifdef TESTING properties and instances for testing #endif ) where Then I just define TESTING when running the test. This has the advantage that the tests at least always compile, even as I make other changes. In reality, FilePath only requires a couple of things (the drive properties) to be exported for testing but not for real use. I don't see much complication in extending my approach. In fact, I suspect that my FilePath test extractor is overly complex, since it allows: Windows: property Posix: property property For Windows properties it added System.FilePath.Windows.functions, ditto for Posix, and for normal ones it generates the test twice, once with Windows and once with Posix. It also detects if a property has any free variables, and if not changes it to a single assertion, rather than a quickcheck property. I think it is very handy, and integrating it with Cabal and removing the special cases that FilePath requires is probably well worth the effort! I would certainly like such a tool... Thanks Neil

Paul Johnson

19 Mar 19 Mar

8:06 p.m.

...

I don't see much complication in extending my approach. In fact, I suspect that my FilePath test extractor is overly complex, since it allows:

Windows: property Posix: property property I actually think that something like this should be preserved. I can imagine several reasons for having flags of this kind. Platform-specific flags are an obvious one. Another would be to distinguish QuickCheck, HUnit and FastCheck tests, as well as tests that take no arguments. Currently Neil treats these as QuickCheck tests, but

Neil Mitchell wrote: this (unless I'm mistaken) will result in each one being called 100 times, which shouldn't be necessary. In the long run we will also need to handle multi-line properties, the QuickCheck forEach combinator, and probably stateful tests of monadic actions. So I suggest that we look for lines of the form -- > q: reverse (xs ++ ys) == -- > (reverse ys ++ reverse xs) -- > u: reverse "foobar" == "raboof" -- > u: null $ reverse [] The letters before the colon are a series of flags, q = QuickCheck, u = Unit, f = FastCheck, W = Windows only, P = Posix only, and so on. Note that the "reverse" example is split across lines. Incidentally, how does the test extractor handle arguments? Does it parse the code for variables? If so then could we have a line to declare argument types for properties where they can't be inferred? Something like: -- > t: xs, ys :: [Int] -- > t: str :: String Would this make sense? Paul.

Neil Mitchell

8:16 p.m.

...

I actually think that something like this should be preserved. I can imagine several reasons for having flags of this kind. Platform-specific flags are an obvious one. Another would be to distinguish QuickCheck, HUnit and FastCheck tests, as well as tests that take no arguments. Currently Neil treats these as QuickCheck tests, but this (unless I'm mistaken) will result in each one being called 100 times, which shouldn't be necessary.

Nope, I treat them as properties and assert them once. The algorithm I use is that things which are single letters are free variables, and single letter followed by 's' - if a property has no free variables its an assertion and only gets executed once.

...

So I suggest that we look for lines of the form

-- > q: reverse (xs ++ ys) == -- > (reverse ys ++ reverse xs) -- > u: reverse "foobar" == "raboof" -- > u: null $ reverse []

I would have thought:

...

-- > forall xs ys . reverse (xs ++ ys) == (reverse ys ++ reverse xs) -- > reverse "foobar" == "raboof" -- > null $ reverse []

I suspect the quantification will be needed explicitly so you can tell between reverse the free variable and reverse the function.

...

Incidentally, how does the test extractor handle arguments? Does it parse the code for variables? If so then could we have a line to declare argument types for properties where they can't be inferred? Something like:

-- > t: xs, ys :: [Int] -- > t: str :: String

I have a scheme where some set of letters refers to file names, and some to other stuff. I also wrap things as well, to get a different quickcheck generator - i.e. newtype CheckFilePath = CheckFilePath FilePath, then unwrap it before doing the actual call. There are lots of design decisions - the best thing would be to set up a wiki page as the theoretical documentation of the tool, then once the manual is stable you can write the tool. Feel free to take any or none of my tool as the starting point. Thanks Neil

Jeffrey Yasskin

1 Apr 1 Apr

6:37 a.m.

I've been working on getting test running Cabalized and finally have a patch to send. Extracting tests from the source code is, I think, an independent problem. http://jeffrey.yasskin.info/darcs/filepath/Setup.hs has the main instructions for running the test (with the same test extraction as the original GenTests.hs), and http://jeffrey.yasskin.info/darcs/filepath/SingleFileOps.hs has some operations that I'd like to add to Cabal itself. With some more experience writing tests like this, we'll definitely want to extract most of this Setup.hs into a shared module. The basic idea is that I want to base the tests as much as possible on production code, but I may need to #define TESTING or some other symbol in a particular file to get that file's tests to run. So each file "foo.hs" may be compiled with the 'tag' "foo" (practically, into the dist/build/foo directory) with -DTESTING, and then its test is run in the context of ["foo", ""] (look for "foo" object files, and then the results of `Setup.hs build`). Putting testing builds in a different place also makes sure that they don't get installed or released accidentally. What would be the best way to proceed? Is this system overkill and I should start over with something simpler (either a single -DTESTING build tree or just `runhaskell -DTESTING`)? Should I clean up SingleFileOps and submit it to Cabal? Should I take what I have and apply it to base? This patch is based on http://www.cs.york.ac.uk/fp/darcs/filepath/, so it's a little out of date. Neil, if you're interested in accepting this for the version of FilePath on darcs.haskell.org, I'll get you a patch that works there in the next week. On 3/18/07, Neil Mitchell wrote:

...

Hi

...
...
This gives a list of properties, which are automatically extracted from the documentation, and checked using QuickCheck. All the tool is in the repo. If anyone wanted to generalise the code in there, it would be handy!

Neil, your extractor looks cool, but I wonder how it'll work with modules like Data.Set whose (commented out) tests define some Arbitrary instances and some helper functions in addition to the actual properties. My first inclination would have been to define the properties in code, perhaps "#ifdef TESTING"ed out, and try to teach Haddock to pull them into the documentation. But yours is definitely simpler and probably worth pursuing until it obviously stops working.

FilePath also has that issue, I do:

module FilePath( #ifdef TESTING properties and instances for testing #endif

) where

Then I just define TESTING when running the test. This has the advantage that the tests at least always compile, even as I make other changes. In reality, FilePath only requires a couple of things (the drive properties) to be exported for testing but not for real use.

I don't see much complication in extending my approach. In fact, I suspect that my FilePath test extractor is overly complex, since it allows:

Windows: property Posix: property property

For Windows properties it added System.FilePath.Windows.functions, ditto for Posix, and for normal ones it generates the test twice, once with Windows and once with Posix. It also detects if a property has any free variables, and if not changes it to a single assertion, rather than a quickcheck property.

I think it is very handy, and integrating it with Cabal and removing the special cases that FilePath requires is probably well worth the effort! I would certainly like such a tool...

Thanks

Neil

-- Namasté, Jeffrey Yasskin

Neil Mitchell

19 Apr 19 Apr

2:29 p.m.

Hi Jeffrey, Sorry for the delay in replying - I've been busy with other stuff.

...

I've been working on getting test running Cabalized and finally have a patch to send. Extracting tests from the source code is, I think, an independent problem.

The code for FilePath is going to be including with GHC 6.6.1 which comes out very shortly, so I'd like to put off adding things to Cabal to run the tests until after that. Every change at this time is a little dangerous so I want to keep them minimal. However, after GHC 6.6.1 comes out, and we have a bit more breathing space, I'd love to have the Setup.hs tests run the tests, rather than having a separate .bat file do it. Ideally I want to be consistent with the other packages, so maybe Ian can advise as to how testing should look like? What does testing need to do to get into the GHC buildbot test scripts?

...

http://jeffrey.yasskin.info/darcs/filepath/SingleFileOps.hs has some operations that I'd like to add to Cabal itself. With some more experience writing tests like this, we'll definitely want to extract most of this Setup.hs into a shared module.

You'll want to look at the Cabal lists for what of the SingleFileOps can go into there.

...

The basic idea is that I want to base the tests as much as possible on production code, but I may need to #define TESTING or some other symbol in a particular file to get that file's tests to run. So each file "foo.hs" may be compiled with the 'tag' "foo" (practically, into the dist/build/foo directory) with -DTESTING, and then its test is run in the context of ["foo", ""] (look for "foo" object files, and then the results of `Setup.hs build`). Putting testing builds in a different place also makes sure that they don't get installed or released accidentally.

All sounds sensible.

...

What would be the best way to proceed? Is this system overkill and I should start over with something simpler (either a single -DTESTING build tree or just `runhaskell -DTESTING`)? Should I clean up SingleFileOps and submit it to Cabal? Should I take what I have and apply it to base?

You'd have to ask the Cabal people how much test stuff they want in Cabal, and perhaps have a separate tool that does test extraction if they'd rather it was kept minimal. Thanks Neil

...

On 3/18/07, Neil Mitchell wrote:

...
Hi

...
...
This gives a list of properties, which are automatically extracted from the documentation, and checked using QuickCheck. All the tool is in the repo. If anyone wanted to generalise the code in there, it would be handy!

Neil, your extractor looks cool, but I wonder how it'll work with modules like Data.Set whose (commented out) tests define some Arbitrary instances and some helper functions in addition to the actual properties. My first inclination would have been to define the properties in code, perhaps "#ifdef TESTING"ed out, and try to teach Haddock to pull them into the documentation. But yours is definitely simpler and probably worth pursuing until it obviously stops working.

FilePath also has that issue, I do:

module FilePath( #ifdef TESTING properties and instances for testing #endif

) where

Then I just define TESTING when running the test. This has the advantage that the tests at least always compile, even as I make other changes. In reality, FilePath only requires a couple of things (the drive properties) to be exported for testing but not for real use.

I don't see much complication in extending my approach. In fact, I suspect that my FilePath test extractor is overly complex, since it allows:

Windows: property Posix: property property

For Windows properties it added System.FilePath.Windows.functions, ditto for Posix, and for normal ones it generates the test twice, once with Windows and once with Posix. It also detects if a property has any free variables, and if not changes it to a single assertion, rather than a quickcheck property.

I think it is very handy, and integrating it with Cabal and removing the special cases that FilePath requires is probably well worth the effort! I would certainly like such a tool...

Thanks

Neil

-- Namasté, Jeffrey Yasskin

Simon Peyton-Jones

19 Mar 19 Mar

8:58 a.m.

Just to say a) I think it's great that you guys are working on a test framework for the libraries b) Simon M is away this week on holiday, so you won't hear from him till next week Simon | -----Original Message----- | From: libraries-bounces@haskell.org [mailto:libraries-bounces@haskell.org] On Behalf Of Jeffrey | Yasskin | Sent: 18 March 2007 01:55 | To: Neil Mitchell | Cc: libraries@haskell.org | Subject: Re: Automated tests for the libraries? | | | | -----Original Message----- | From: libraries-bounces@haskell.org [mailto:libraries-bounces@haskell.org] On Behalf Of Jeffrey | Yasskin | Sent: 18 March 2007 01:55 | To: Neil Mitchell | Cc: libraries@haskell.org | Subject: Re: Automated tests for the libraries? | | On 3/16/07, Neil Mitchell wrote: | > Hi Paul | > | > > I think that QuickCheck properties, or at least an abbreviated version | > > of them, should be in the documentation. See my RangedSet library | > > (http://ranged-sets.sourceforge.net/) for an example of the kind of | > > thing I mean. However I have to maintain the correspondence between the | > > actual properties and the Haddock comments manually, which is a pain. | > | > See my FilePath library: | > http://www-users.cs.york.ac.uk/~ndm/projects/libraries.php | > | > For example: http://www-users.cs.york.ac.uk/~ndm/projects/filepath/System-FilePath- | Version_0_11.html#v%3AreplaceExtension | > | > This gives a list of properties, which are automatically extracted | > from the documentation, and checked using QuickCheck. All the tool is | > in the repo. If anyone wanted to generalise the code in there, it | > would be handy! | > | > Thanks | | Neat! I've started http://haskell.org/haskellwiki/Library_tests linked | from Library_submissions to record this stuff. Feel free to rearrange | that of course. | | The biggest problem I see with the tests that SimonM is moving is that | they have to be run inside a built GHC source tree, which makes it | hard to work just on the libraries or test them on another compiler. | | Neil, your extractor looks cool, but I wonder how it'll work with | modules like Data.Set whose (commented out) tests define some | Arbitrary instances and some helper functions in addition to the | actual properties. My first inclination would have been to define the | properties in code, perhaps "#ifdef TESTING"ed out, and try to teach | Haddock to pull them into the documentation. But yours is definitely | simpler and probably worth pursuing until it obviously stops working. | | It would be nice to use Cabal to run these tests since it already | knows how to pick a compiler. I'll start on a runTests hook using | GenTests.hs. | | -- | Namasté, | Jeffrey Yasskin

Ian Lynagh

21 Mar 21 Mar

12:04 a.m.

On Sat, Mar 17, 2007 at 06:54:57PM -0700, Jeffrey Yasskin wrote:

...

The biggest problem I see with the tests that SimonM is moving is that they have to be run inside a built GHC source tree, which makes it hard to work just on the libraries or test them on another compiler.

This is really a problem with the driver and infrastructure, not the tests themselves. It would be good to have a standalone driver program that doesn't depend on the hierarchy of Makefiles and knows how to run the tests with different implementations. Thanks Ian

Jeffrey Yasskin

19 Apr 19 Apr

3:49 p.m.

On 3/20/07, Ian Lynagh wrote:

...

On Sat, Mar 17, 2007 at 06:54:57PM -0700, Jeffrey Yasskin wrote:

...
The biggest problem I see with the tests that SimonM is moving is that they have to be run inside a built GHC source tree, which makes it hard to work just on the libraries or test them on another compiler.

This is really a problem with the driver and infrastructure, not the tests themselves. It would be good to have a standalone driver program that doesn't depend on the hierarchy of Makefiles and knows how to run the tests with different implementations.

To that end, I've written the attached patch. It keeps runtests.py embedded in the ghc testsuite, but you no longer have to run it from the Makefiles. From the libraries/unix/tests directory, both of $ ../../../testsuite/driver/runtests.py --compiler=`which ghc` $ ../../../testsuite/driver/runtests.py --compiler=../../../compiler/stage1/ghc-inplace pass, although they only run the normal and optc ways. I haven't updated the hugs config or written a config for any other compiler. Jeffrey

6653

Age (days ago)

6687

Last active (days ago)

List overview

Download

13 comments

6 participants

participants (6)

dons＠cse.unsw.edu.au
Ian Lynagh
Jeffrey Yasskin
Neil Mitchell
Paul Johnson
Simon Peyton-Jones