Hi Sean. Thanks a bunch for these tips. I haven't used the flags feature of cabal before, and i don't seem to be able to get it right. I have: Flag test Description: Enable testing Default: False And I get "Warning: unamb.cabal: Unknown section type: flag ignoring...". If I indent, I instead get "These flags are used without having been defined: test". Any idea what I'm doing wrong here? - Conal On Tue, Sep 9, 2008 at 4:32 PM, Sean Leather wrote:

How do folks like to package up QuickCheck tests for their libraries? In

...

...

...

the main library? As a separate repo & package? Same repo & separate package? Keeping tests with the tested code allows testing of non-exported functionality, but can add quite a lot of clutter.

I have QuickCheck properties plus HUnit tests, but I think the question is the same. For me, it's in the same repository and shipped with the package source. I think that if you ship source (even via Hackage), you should also ship tests. So, if somebody wants to modify the source, they can run the tests. And making it convenient to test is very important, so I have "cabal test" (or "runhaskell Setup.hs test" without cabal-install) configured to run the tests. I don't think tests should (in general) be part of the user-visible API, so I have them external to the module hierarchy.

How do you set up cabal to do these tests?

I use the "runTests" hook in Distribution.Simple. The code below works on Windows and Mac, because that's what we use.

\begin{code} module Main (main) where

import Distribution.Simple import System.Cmd (system) import System.FilePath (())

main :: IO () main = defaultMainWithHooks hooks where hooks = simpleUserHooks { runTests = runTests' }

runTests' _ _ _ _ = system cmd >> return () where testdir = "dist" "build" "test" testcmd = "." "test" cmd = "cd " ++ testdir ++ " && " ++ testcmd \end{code}

Do your libraries depend on HUnit?

...

No, because I use an ultra-secret trick. ;) I have a Library in my .cabal file and an Executable for testing. Part of the test description follows.

\begin{cabal} Executable test hs-source-dirs: src, tests, examples main-is: Main.hs

-- Only enable the build-depends here if configured with "-ftest". This -- keeps users from having to install QuickCheck 2 in order to use EMGM. if flag(test) build-depends: QuickCheck >= 2.0, HUnit >= 1.2 else buildable: False \end{cabal}

With that last flag-based if/else, I hide the dependencies for normal building ('test' by default is False). If 'test' is False, then the executable also cannot be built.

Where do you like to place your tests? In the functionality modules? A

...

parallel structure? A single Test.hs file somewhere?

In a separate "tests" directory at the same level as the "src" directory containing the module hierarchy. It has a number of files, mostly one per module tested.

...

Testing non-exported functionality without exporting the test interface

...

seems difficult in general. Is there a way to hide part of a module interface with Cabal? Then, you could have a 'test' function exported from each module for testing but hidden for release.

My current leaning is to split a package "foo" into packages "foo" and

...

...

"foo-test"

What benefit does this provide?

It keeps the library and its dependencies small. Probably some of the alternatives do as well. For testing, I'm using checkershttp://haskell.org/haskellwiki/Checkersin addition to QuickCheck, and I'd prefer not to make casual library users have to pull in those libraries as well.

Ah, so you're handling the same problem we are in a different way. Nice!

Sean

Ketil Malde wrote:

"Conal Elliott" writes:

...

Thanks a bunch for these tips. I haven't used the flags feature of cabal before, and i don't seem to be able to get it right.

Another option might be to have the test command build via 'ghc --make' instead of Cabal - this way, you can avoid mentioning testing libraries altogether in the cabal file.

This is the approach I have often taken in the past, but it imposes somewhat of a maintenance burden because you mus tthen make sure that the ghc command line flags are kept in-sync what what you're doing in the cabal file -- -X options, packages used, etc. This becomes even more difficult if your cabal file is doing any sort of dynamic configuration.

My tests are making use of a nice console test runner I wrote that supports both HUnit and QuickCheck (and is extensible to other test providers by the user): http://hackage.haskell.org/cgi-bin/hackage-scripts/package/test-framework.

The description looks great! I might have to try it out. I used HUnit with QuickCheck 2, so that I could run QC properties as HUnit tests. QC2 has the added ability (over QC1) to run a property and return a Bool instead of just exiting with an error, and that works nicely within HUnit. Does test-framework do something else to support QC running side-by-side with HUnit? Sean

Sean Leather wrote:

My tests are making use of a nice console test runner I wrote that supports both HUnit and QuickCheck (and is extensible to other test providers by the user): http://hackage.haskell.org/cgi-bin/hackage-scripts/package/test-framework.

The description looks great! I might have to try it out.

I used HUnit with QuickCheck 2, so that I could run QC properties as HUnit tests. QC2 has the added ability (over QC1) to run a property and return a Bool instead of just exiting with an error, and that works nicely within HUnit. Does test-framework do something else to support QC running side-by-side with HUnit?

See: http://software.complete.org/static/missingh/doc/MissingH/Test-HUnit-Utils.h... Also some examples at http://git.complete.org/offlineimap?a=tree;f=testsrc;h=217ee16404384ba2ae3ad... see runtests.hs and TestInfrastructure.hs

2008/9/9 Conal Elliott :

Where do you like to place your tests? In the functionality modules? A parallel structure? A single Test.hs file somewhere?

The last time I had a chance to experiment with how to do this I used a single Test.hs for the whole project and I think that is a bad idea now. I agree that you don't want to clutter your code with the test cases. While it's good to have the tests accessible and near to the actual code it can be distracting too. The approach I used is here: http://blog.codersbase.com/2006/09/01/simple-unit-testing-in-haskell/ Basically, I used the H98 parser plus TH to collect all the test cases together and generate the test harness at compile time. This meant that any module that had a test case had to be plain H98 (but again, only Test.hs had test cases). On the other hand, specifying tests was as simple as starting a function name with "prop_". You could also modify my technique to look through all modules, or modules with specific names. Another approach is to use conditional compilation with something like CPP. This could allow you to export everything from modules only while testing and then you could have Foo.hs and FooTests.hs for every module. The latter one would import everything from Foo.hs and define the tests. If you don't like conditional compilation, another approach I think is decent is to have FooPrivate.hs (or FooInternal.hs), which is imported into Foo.hs and FooTests.hs. You could set things up so that only Foo.hs and FooTests.hs are allowed to import FooPrivate.hs. You could of course write a script to enforce this, but something that tends to be simpler and equally effective is just to politely ask that people do not import FooPrivate.hs except in FooTests.hs and Foo.hs. I hope that helps, Jason

Hello Johannes, Wednesday, September 10, 2008, 9:39:15 AM, you wrote:

Has there ever been a discussion of typed, user-definable, user-processable source code annotations for Haskell?

afair it was on haskell-prime list btw, Template Haskell may be used for this purpose (although not in portable way, of course) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com

Am Dienstag, 9. September 2008 16:05 schrieb Conal Elliott:

[…]

...

...

My current leaning is to split a package "foo" into packages "foo" and "foo-test"

What benefit does this provide?

It keeps the library and its dependencies small.

Do you publish foo-test on Hackage? If yes than the HackageDB gets cluttered with test packages. If not, there is no easy way for users to run your tests.

[…]

Best wishes, Wolfgang