
I've been picking up Haskell as a side project for the last few months, and I'm now considering publishing some useful code I've written and reused in several small projects. So far I've done relatively little with testing (these have been non-mission-critical applications) but I feel I should get my act together before pushing my work into the common ecosystem. I'm comfortable writing tests in QuickCheck and HUnit and bundling them as optional executables with cabal, but I understand there's a better way. Specifically, I'm looking at the test-framework package and cabal's (newish) test-suite sections. Are these two used together or each to the exclusion of the other? Is there something else I should be using? Cheers, Mike Craig

On Thu, Feb 2, 2012 at 4:30 PM, Michael Craig
I've been picking up Haskell as a side project for the last few months, and I'm now considering publishing some useful code I've written and reused in several small projects. So far I've done relatively little with testing (these have been non-mission-critical applications) but I feel I should get my act together before pushing my work into the common ecosystem.
I'm comfortable writing tests in QuickCheck and HUnit and bundling them as optional executables with cabal, but I understand there's a better way. Specifically, I'm looking at the test-framework package and cabal's (newish) test-suite sections. Are these two used together or each to the exclusion of the other? Is there something else I should be using?
I use Cabal's test-suite support to make building and running a test executable easy, and I use test-framework to write my 'main' method and provide nice output to the user. They have worked well together for me. Antoine

On 3 February 2012 06:30, Michael Craig
I'm comfortable writing tests in QuickCheck and HUnit and bundling them as optional executables with cabal, but I understand there's a better way. Specifically, I'm looking at the test-framework package and cabal's (newish) test-suite sections. Are these two used together or each to the exclusion of the other? Is there something else I should be using?
I've followed what Johan Tibbell did in the hashable package: http://hackage.haskell.org/packages/archive/hashable/1.1.2.2/hashable.cabal You can see that it depends on test-framework and test-framework-quickcheck2 packages, and uses these in a cabal Test-suite stanza. It works pretty well for me (in zoom-cache). Note that you need to "cabal configure --enable-tests" before building your package and running "cabal test". cheers, Conrad.

On Thu, Feb 2, 2012 at 4:19 PM, Conrad Parker
I've followed what Johan Tibbell did in the hashable package:
If I had known how much confusion my childhood friends would unleash on the Internet when they, at age 7, gave me a nickname that's spelled slightly differently from my last name, I would have asked them to pick another one. ;) -- Johan Tibell

On 3 February 2012 08:30, Johan Tibell
On Thu, Feb 2, 2012 at 4:19 PM, Conrad Parker
wrote: I've followed what Johan Tibbell did in the hashable package:
If I had known how much confusion my childhood friends would unleash on the Internet when they, at age 7, gave me a nickname that's spelled slightly differently from my last name, I would have asked them to pick another one. ;)
lol, sorry, I actually double-checked the number of l's before writing that but didn't consider the b's. For future reference I've produced a handy chart: Letter | Real-name count | Nickname count -------+-----------------+--------------- b | 1 | 2 l | 2 | 0 -------+-----------------+--------------- SUM | 3 | 2 K.

On Thu, Feb 2, 2012 at 4:46 PM, Conrad Parker
On Thu, Feb 2, 2012 at 4:19 PM, Conrad Parker
wrote: I've followed what Johan Tibbell did in the hashable package:
If I had known how much confusion my childhood friends would unleash on
On 3 February 2012 08:30, Johan Tibell
wrote: the Internet when they, at age 7, gave me a nickname that's spelled slightly differently from my last name, I would have asked them to pick another one. ;)
lol, sorry, I actually double-checked the number of l's before writing that but didn't consider the b's. For future reference I've produced a handy chart:
Letter | Real-name count | Nickname count -------+-----------------+--------------- b | 1 | 2 l | 2 | 0 -------+-----------------+--------------- SUM | 3 | 2
Excellent. I will tattoo it on my forehead.

On 03/02/2012 12:22 PM, "Johan Tibell"
On Thu, Feb 2, 2012 at 4:46 PM, Conrad Parker
wrote:
On 3 February 2012 08:30, Johan Tibell
wrote: On Thu, Feb 2, 2012 at 4:19 PM, Conrad Parker
wrote:
I've followed what Johan Tibbell did in the hashable package:
If I had known how much confusion my childhood friends would unleash on the Internet when they, at age 7, gave me a nickname that's spelled slightly differently from my last name, I would have asked them to pick another one. ;)
lol, sorry, I actually double-checked the number of l's before writing that but didn't consider the b's. For future reference I've produced a handy chart:
Letter | Real-name count | Nickname count -------+-----------------+--------------- b | 1 | 2 l | 2 | 0 -------+-----------------+--------------- SUM | 3 | 2
Excellent. I will tattoo it on my forehead.
There is, of course, a simpler (but not necessarily easier :p) solution: change your name to match your nickname!
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Thanks for the advice, all. I've got test-framework, quickcheck, and cabal's test-suite all working together nicely. Cabal seems to support using hpc to check test coverage. If I add -fhpc to the ghc-options under the test-suite, I get output like "Test coverage report written to dist/hpc/html/tests/hpc_index.html" and "Package coverage report written to dist/hpc/html/test-0.0.0/hpc_index.html", but those html files are just empty tables. How does this work? Mike Craig On Thu, Feb 2, 2012 at 8:45 PM, Ivan Lazar Miljenovic < ivan.miljenovic@gmail.com> wrote:
On 03/02/2012 12:22 PM, "Johan Tibell"
wrote: On Thu, Feb 2, 2012 at 4:46 PM, Conrad Parker
wrote:
On 3 February 2012 08:30, Johan Tibell
wrote: On Thu, Feb 2, 2012 at 4:19 PM, Conrad Parker
wrote:
I've followed what Johan Tibbell did in the hashable package:
If I had known how much confusion my childhood friends would unleash on the Internet when they, at age 7, gave me a nickname that's spelled slightly differently from my last name, I would have asked them to pick another one. ;)
lol, sorry, I actually double-checked the number of l's before writing that but didn't consider the b's. For future reference I've produced a handy chart:
Letter | Real-name count | Nickname count -------+-----------------+--------------- b | 1 | 2 l | 2 | 0 -------+-----------------+--------------- SUM | 3 | 2
Excellent. I will tattoo it on my forehead.
There is, of course, a simpler (but not necessarily easier :p) solution: change your name to match your nickname!
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

If you're writing a library, you need to compile the library with
`-fhpc`, i.e. put it in the library stanza, not the testsuite stanza,
and then you can compile the test program using your library - the
resulting 'tix' file will contain the library coverage reports. You
can link a HPC-built library into an executable not compiled with HPC
just fine.
Normally I only compile the library under HPC mode, link it in a test,
and distribute the results from that. That way your coverage reports
don't include the test module (which may or may not be relevant.)
I normally add a cabal flag called 'hpc' which optionally enables
coverage reports for my library, e.g.
flag hpc
default: False
library
...
...
if flag(hpc)
ghc-options: -fhpc
Then when you want coverage reports, just say 'cabal install -fhpc
--enable-tests' and the resulting properties executable will spit out
the results when run.
On Tue, Feb 7, 2012 at 3:16 PM, Michael Craig
Thanks for the advice, all. I've got test-framework, quickcheck, and cabal's test-suite all working together nicely.
Cabal seems to support using hpc to check test coverage. If I add -fhpc to the ghc-options under the test-suite, I get output like "Test coverage report written to dist/hpc/html/tests/hpc_index.html" and "Package coverage report written to dist/hpc/html/test-0.0.0/hpc_index.html", but those html files are just empty tables. How does this work?
Mike Craig
On Thu, Feb 2, 2012 at 8:45 PM, Ivan Lazar Miljenovic
wrote: On 03/02/2012 12:22 PM, "Johan Tibell"
wrote: On Thu, Feb 2, 2012 at 4:46 PM, Conrad Parker
wrote: On 3 February 2012 08:30, Johan Tibell
wrote: On Thu, Feb 2, 2012 at 4:19 PM, Conrad Parker
wrote: I've followed what Johan Tibbell did in the hashable package:
If I had known how much confusion my childhood friends would unleash on the Internet when they, at age 7, gave me a nickname that's spelled slightly differently from my last name, I would have asked them to pick another one. ;)
lol, sorry, I actually double-checked the number of l's before writing that but didn't consider the b's. For future reference I've produced a handy chart:
Letter | Real-name count | Nickname count -------+-----------------+--------------- b | 1 | 2 l | 2 | 0 -------+-----------------+--------------- SUM | 3 | 2
Excellent. I will tattoo it on my forehead.
There is, of course, a simpler (but not necessarily easier :p) solution: change your name to match your nickname!
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
-- Regards, Austin

By the way, has anyone else had trouble with "cabal test" diverging?
I've been running into this issue with cabal 0.10.2, but ONLY in
conjunction with GHC 6.12.3. It's hard to make a small reproducer for (and
therefore I haven't filed a bug yet), but you can see the below Jenkins run
stalled for 2.5 days, whereas it should take minutes:
http://tester-lin.soic.indiana.edu:8080/job/monad-par_github_master/JENKINS_...
Note that it *doesn't* burn CPU -- it deadlocks rather than spins.
I replaced "cabal test" with a direct call to the test executable and I
haven't seen this problem since.
-Ryan
On Tue, Feb 7, 2012 at 4:23 PM, Austin Seipp
If you're writing a library, you need to compile the library with `-fhpc`, i.e. put it in the library stanza, not the testsuite stanza, and then you can compile the test program using your library - the resulting 'tix' file will contain the library coverage reports. You can link a HPC-built library into an executable not compiled with HPC just fine.
Normally I only compile the library under HPC mode, link it in a test, and distribute the results from that. That way your coverage reports don't include the test module (which may or may not be relevant.)
I normally add a cabal flag called 'hpc' which optionally enables coverage reports for my library, e.g.
flag hpc default: False
library ... ... if flag(hpc) ghc-options: -fhpc
Then when you want coverage reports, just say 'cabal install -fhpc --enable-tests' and the resulting properties executable will spit out the results when run.
Thanks for the advice, all. I've got test-framework, quickcheck, and cabal's test-suite all working together nicely.
Cabal seems to support using hpc to check test coverage. If I add -fhpc to the ghc-options under the test-suite, I get output like "Test coverage report written to dist/hpc/html/tests/hpc_index.html" and "Package coverage report written to dist/hpc/html/test-0.0.0/hpc_index.html", but those
On Tue, Feb 7, 2012 at 3:16 PM, Michael Craig
wrote: html files are just empty tables. How does this work?
Mike Craig
On Thu, Feb 2, 2012 at 8:45 PM, Ivan Lazar Miljenovic
wrote: On 03/02/2012 12:22 PM, "Johan Tibell"
wrote: On Thu, Feb 2, 2012 at 4:46 PM, Conrad Parker
wrote: On 3 February 2012 08:30, Johan Tibell
wrote:
On Thu, Feb 2, 2012 at 4:19 PM, Conrad Parker < conrad@metadecks.org> wrote: > > I've followed what Johan Tibbell did in the hashable package:
If I had known how much confusion my childhood friends would unleash on the Internet when they, at age 7, gave me a nickname that's spelled slightly differently from my last name, I would have asked them to pick another one. ;)
lol, sorry, I actually double-checked the number of l's before writing that but didn't consider the b's. For future reference I've produced a handy chart:
Letter | Real-name count | Nickname count -------+-----------------+--------------- b | 1 | 2 l | 2 | 0 -------+-----------------+--------------- SUM | 3 | 2
Excellent. I will tattoo it on my forehead.
There is, of course, a simpler (but not necessarily easier :p) solution: change your name to match your nickname!
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
-- Regards, Austin
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

How do you control where the .mix files are generated? GHC is putting them
in .hpc/, but I'm getting errors like "hpc: can not find
test-0.0.0/Data.Test in ["./dist/hpc/mix/test-0.0.0"]".
Mike Craig
On Tue, Feb 7, 2012 at 5:12 PM, Ryan Newton
By the way, has anyone else had trouble with "cabal test" diverging?
I've been running into this issue with cabal 0.10.2, but ONLY in conjunction with GHC 6.12.3. It's hard to make a small reproducer for (and therefore I haven't filed a bug yet), but you can see the below Jenkins run stalled for 2.5 days, whereas it should take minutes:
http://tester-lin.soic.indiana.edu:8080/job/monad-par_github_master/JENKINS_...
Note that it *doesn't* burn CPU -- it deadlocks rather than spins.
I replaced "cabal test" with a direct call to the test executable and I haven't seen this problem since.
-Ryan
On Tue, Feb 7, 2012 at 4:23 PM, Austin Seipp
wrote: If you're writing a library, you need to compile the library with `-fhpc`, i.e. put it in the library stanza, not the testsuite stanza, and then you can compile the test program using your library - the resulting 'tix' file will contain the library coverage reports. You can link a HPC-built library into an executable not compiled with HPC just fine.
Normally I only compile the library under HPC mode, link it in a test, and distribute the results from that. That way your coverage reports don't include the test module (which may or may not be relevant.)
I normally add a cabal flag called 'hpc' which optionally enables coverage reports for my library, e.g.
flag hpc default: False
library ... ... if flag(hpc) ghc-options: -fhpc
Then when you want coverage reports, just say 'cabal install -fhpc --enable-tests' and the resulting properties executable will spit out the results when run.
Thanks for the advice, all. I've got test-framework, quickcheck, and cabal's test-suite all working together nicely.
Cabal seems to support using hpc to check test coverage. If I add -fhpc to the ghc-options under the test-suite, I get output like "Test coverage report written to dist/hpc/html/tests/hpc_index.html" and "Package coverage report written to dist/hpc/html/test-0.0.0/hpc_index.html", but those
files are just empty tables. How does this work?
Mike Craig
On Thu, Feb 2, 2012 at 8:45 PM, Ivan Lazar Miljenovic
wrote: On 03/02/2012 12:22 PM, "Johan Tibell"
wrote: On Thu, Feb 2, 2012 at 4:46 PM, Conrad Parker
wrote: On 3 February 2012 08:30, Johan Tibell
wrote:
> On Thu, Feb 2, 2012 at 4:19 PM, Conrad Parker < conrad@metadecks.org> > wrote: >> >> I've followed what Johan Tibbell did in the hashable package: > > > If I had known how much confusion my childhood friends would unleash > on the > Internet when they, at age 7, gave me a nickname that's spelled > slightly > differently from my last name, I would have asked them to pick > another one. > ;)
lol, sorry, I actually double-checked the number of l's before writing that but didn't consider the b's. For future reference I've
On Tue, Feb 7, 2012 at 3:16 PM, Michael Craig
wrote: html produced a handy chart:
Letter | Real-name count | Nickname count -------+-----------------+--------------- b | 1 | 2 l | 2 | 0 -------+-----------------+--------------- SUM | 3 | 2
Excellent. I will tattoo it on my forehead.
There is, of course, a simpler (but not necessarily easier :p) solution: change your name to match your nickname!
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
-- Regards, Austin
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

On Tue, Feb 7, 2012 at 3:23 PM, Austin Seipp
If you're writing a library, you need to compile the library with `-fhpc`, i.e. put it in the library stanza, not the testsuite stanza, and then you can compile the test program using your library - the resulting 'tix' file will contain the library coverage reports. You can link a HPC-built library into an executable not compiled with HPC just fine.
Normally I only compile the library under HPC mode, link it in a test, and distribute the results from that. That way your coverage reports don't include the test module (which may or may not be relevant.)
I normally add a cabal flag called 'hpc' which optionally enables coverage reports for my library, e.g.
flag hpc default: False
library ... ... if flag(hpc) ghc-options: -fhpc
Then when you want coverage reports, just say 'cabal install -fhpc --enable-tests' and the resulting properties executable will spit out the results when run.
First, as author of the test suite code, let me apologize for the terrible documentation. This is absolutely NOT how coverage reports are supposed to work. If you configure with '--enable-tests --enable-library-coverage', Cabal will take care of _everything_ else for you, including excluding the test module from the coverage report. You should not have to put any flag gymnastics in your .cabal file. If this doesn't work for you, please let me know, because it's a bug. Thanks! -- Thomas Tuegel

Hello Thomas,
On Wed, Feb 8, 2012 at 4:03 AM, Thomas Tuegel
First, as author of the test suite code, let me apologize for the terrible documentation.
This is absolutely NOT how coverage reports are supposed to work. If you configure with '--enable-tests --enable-library-coverage', Cabal will take care of _everything_ else for you, including excluding the test module from the coverage report. You should not have to put any flag gymnastics in your .cabal file.
I get only "cabal: unrecognized option `--enable-library-coverage'" when doing that. Which cabal version do I need? Can you point me to any docs about this? I'm still using test type exitcode-stdio-1.0, because I could not get detailed-1.0 to work. TIA, Chris

On Wed, Feb 8, 2012 at 4:42 AM, Christoph Breitkopf
Hello Thomas,
On Wed, Feb 8, 2012 at 4:03 AM, Thomas Tuegel
wrote: First, as author of the test suite code, let me apologize for the terrible documentation.
This is absolutely NOT how coverage reports are supposed to work. If you configure with '--enable-tests --enable-library-coverage', Cabal will take care of _everything_ else for you, including excluding the test module from the coverage report. You should not have to put any flag gymnastics in your .cabal file.
I get only "cabal: unrecognized option `--enable-library-coverage'" when doing that. Which cabal version do I need? Can you point me to any docs about this? I'm still using test type exitcode-stdio-1.0, because I could not get detailed-1.0 to work.
You need Cabal 1.12 for '--enable-library-coverage'. The only place it's documented is in 'cabal configure --help' (a major oversight on my part). The online docs for Cabal are only from version 1.10 anyway, so that wouldn't have helped. I'm sorry for your struggles getting the detailed type to work. It's a known issue: the detailed test interface is intentionally disabled in all the released versions because it's going to change. That's why none of the test libraries support it yet and why you weren't able to get it to work. -- Thomas Tuegel

On Thu, Feb 9, 2012 at 6:57 PM, Thomas Tuegel
You need Cabal 1.12 for '--enable-library-coverage'. The only place it's documented is in 'cabal configure --help' (a major oversight on my part). The online docs for Cabal are only from version 1.10 anyway, so that wouldn't have helped.
I'm sorry for your struggles getting the detailed type to work. It's a known issue: the detailed test interface is intentionally disabled in all the released versions because it's going to change. That's why none of the test libraries support it yet and why you weren't able to get it to work.
No problem - I was just wondering where my error lay. The stdio tests do the trick for the time being. - Chris

Thanks Thomas, that new flag is great.
Mike Craig
On Tue, Feb 7, 2012 at 10:03 PM, Thomas Tuegel
On Tue, Feb 7, 2012 at 3:23 PM, Austin Seipp
wrote: If you're writing a library, you need to compile the library with `-fhpc`, i.e. put it in the library stanza, not the testsuite stanza, and then you can compile the test program using your library - the resulting 'tix' file will contain the library coverage reports. You can link a HPC-built library into an executable not compiled with HPC just fine.
Normally I only compile the library under HPC mode, link it in a test, and distribute the results from that. That way your coverage reports don't include the test module (which may or may not be relevant.)
I normally add a cabal flag called 'hpc' which optionally enables coverage reports for my library, e.g.
flag hpc default: False
library ... ... if flag(hpc) ghc-options: -fhpc
Then when you want coverage reports, just say 'cabal install -fhpc --enable-tests' and the resulting properties executable will spit out the results when run.
First, as author of the test suite code, let me apologize for the terrible documentation.
This is absolutely NOT how coverage reports are supposed to work. If you configure with '--enable-tests --enable-library-coverage', Cabal will take care of _everything_ else for you, including excluding the test module from the coverage report. You should not have to put any flag gymnastics in your .cabal file.
If this doesn't work for you, please let me know, because it's a bug.
Thanks!
-- Thomas Tuegel
participants (9)
-
Antoine Latter
-
Austin Seipp
-
Christoph Breitkopf
-
Conrad Parker
-
Ivan Lazar Miljenovic
-
Johan Tibell
-
Michael Craig
-
Ryan Newton
-
Thomas Tuegel