[Hackage] #215: Overhaul support for package's tests
#215: Overhaul support for package's tests ------------------------------+--------------------------------------------- Reporter: duncan | Owner: Type: task | Status: new Priority: normal | Milestone: Component: Cabal library | Version: 1.2.3.0 Severity: normal | Keywords: Difficulty: project(> week) | Ghcversion: 6.8.2 Platform: | ------------------------------+--------------------------------------------- The current support for running package's testsuites is almost uniformly unused and packages instead implement ad-hoc solutions which makes it hard for users to run the testsuites and impossible to automate on hackage. The current support is so trivial anyone who takes this task on can ignore it and start with a clean slate. A [http://www.haskell.org/pipermail/cabal-devel/2008-January/001661.html thread on the issue]. Here are some ideas for possible requirements: * Support tests even in the `Simple` build system. Do not require using the `build-type: Custom` just to be able to use tests. * Define a standard interface for returning test results from a testsuite to Cabal so they can be included into a build report and could be published on the hackage website or some other custom in-house test server. * Support for above interface in `QuickCheck` and `HUnit`. * Support for HPC when running testsuite programs, and gathering coverage results, again for reporting purposes. * We want to be able to run tests on a library that is not installed yet. This will rely on support in Cabal for registering a package inplace in a local package db, and building test code against that. * What about performance tests? What format should that use for reporting? The emphasis is on automation. A lot of interesting information becomes available with enough automation. For example we could compare on hackage which packages have more comprehensive testsuites (measured by HPC coverage). Or we could track performance over time. Once the data is available there is a lot we could do with it. It's not just for a public hackage server, developers could use this privately too. -- Ticket URL: http://hackage.haskell.org/trac/hackage/ticket/215 Hackage http://haskell.org/cabal/ Hackage: Cabal and related projects
#215: Overhaul support for package's tests ----------------------------+----------------------------------------------- Reporter: duncan | Owner: Type: task | Status: new Priority: normal | Milestone: Component: Cabal library | Version: 1.2.3.0 Severity: normal | Resolution: Keywords: | Difficulty: project(> week) Ghcversion: 6.8.2 | Platform: ----------------------------+----------------------------------------------- Changes (by dons): * cc: dons@galois.com (added) -- Ticket URL: http://hackage.haskell.org/trac/hackage/ticket/215#comment:1 Hackage http://haskell.org/cabal/ Hackage: Cabal and related projects
#215: Overhaul support for package's tests ----------------------------+----------------------------------------------- Reporter: duncan | Owner: Type: task | Status: new Priority: normal | Milestone: Component: Cabal library | Version: 1.2.3.0 Severity: normal | Resolution: Keywords: | Difficulty: project(> week) Ghcversion: 6.8.2 | Platform: ----------------------------+----------------------------------------------- Comment (by duncan): Here's an idea {{{ test test-is: Test build-depends: QuickCheck hs-source-dirs: tests src }}} So imaging a dir layout like: {{{ foo.cabal src/* tests/Test.hs }}} where `tests/Test.hs` is {{{ module Test (tests) where tests :: SomethingYetToBeDecided tests = ... }}} So a test is really like an executable or library. It can have all the ordinary buildinfo which for example allows it to depend on extra testing libs. The difference is that instead of having a single Main.main entry point or a bunch of exposed modules, it has a test entry point in one module. That also means the interface is a Haskell one, not a process text stdin/stdout one. That's likely more portable between Haskell implementations and crucially it allows us to build the test code with additional profiling or hpc or whatever. It also means that in principle such tests could be secure if the test type does not involve IO and we use a Haskell implementation that can run untrusted pure code securely. The tricky bit is what the type of the test entry point should be. We want to be able to gather the results of the tests. Perhaps the test stanza also needs to declare what version of the test api interface it is using since it is likely to be refined/improved over time. -- Ticket URL: http://hackage.haskell.org/trac/hackage/ticket/215#comment:2 Hackage http://haskell.org/cabal/ Hackage: Cabal and related projects
#215: Overhaul support for package's tests ----------------------------+----------------------------------------------- Reporter: duncan | Owner: Type: task | Status: new Priority: normal | Milestone: Component: Cabal library | Version: 1.2.3.0 Severity: normal | Resolution: Keywords: | Difficulty: project(> week) Ghcversion: 6.8.2 | Platform: ----------------------------+----------------------------------------------- Comment (by guest):
That also means the interface is a Haskell one, not a process text stdin/stdout one. That's likely more portable between Haskell implementations and crucially it allows us to build the test code with additional profiling or hpc or whatever.
Really? Is it not possible to build with hpc style stuff if the interface is just a command line program? I'm also not overly convinced by the argument about portability. Why not assume Test-Is: Test, means that Test is a module Main, with main in it. You run the tests (perhaps supplying test-arguments: for command line arguments), and the error code is the number of tests fail, where exitSuccess is equivalent to no failures. The log of the test run could also be displayed on hackage. Command line programs are simple, and can be written without learning a new API. They can also be invoked from outside Cabal. For example, the tagsoup program has a tagsoup binary, which has several example programs in it. One of those example things is "tagsoup test" which runs the test properties. Changing to have Test call system "tagsoup test" would be a bit of a shame. In general, something does need to be done, but keeping it very simple seems like a good idea. The idea of having a test program be an actual program with an error code seems to appeal to the Unix ideal of keeping things simple and self-contained. I know for certain that Colin would back me up on this :-) I would also suggest performance testing be given a back seat. In general, its hard to do it well, and get meaningful results. A separate speed-check tool is being worked on, and that can be done separately from Cabal. If it was anyone other than Dons/Duncan having this discussion, no one would have considered performance :-) -- Ticket URL: http://hackage.haskell.org/trac/hackage/ticket/215#comment:3 Hackage http://haskell.org/cabal/ Hackage: Cabal and related projects
#215: Overhaul support for package's tests ----------------------------+----------------------------------------------- Reporter: duncan | Owner: Type: task | Status: new Priority: normal | Milestone: Cabal-1.6 Component: Cabal library | Version: Severity: normal | Resolution: Keywords: | Difficulty: project(> week) Ghcversion: 6.8.2 | Platform: ----------------------------+----------------------------------------------- Changes (by duncan): * version: 1.2.3.0 => * milestone: => Cabal-1.6 -- Ticket URL: http://hackage.haskell.org/trac/hackage/ticket/215#comment:4 Hackage http://haskell.org/cabal/ Hackage: Cabal and related projects
#215: Overhaul support for package's tests ----------------------------+----------------------------------------------- Reporter: duncan | Owner: Type: task | Status: new Priority: normal | Milestone: Cabal-1.8 Component: Cabal library | Version: Severity: normal | Resolution: Keywords: | Difficulty: project(> week) Ghcversion: 6.8.2 | Platform: ----------------------------+----------------------------------------------- Changes (by duncan): * milestone: Cabal-1.6 => Cabal-1.8 -- Ticket URL: http://hackage.haskell.org/trac/hackage/ticket/215#comment:5 Hackage http://haskell.org/cabal/ Hackage: Cabal and related projects
#215: Overhaul support for package's tests ----------------------------+----------------------------------------------- Reporter: duncan | Owner: Type: task | Status: new Priority: normal | Milestone: Cabal-1.8 Component: Cabal library | Version: Severity: normal | Resolution: Keywords: | Difficulty: project(> week) Ghcversion: 6.8.2 | Platform: ----------------------------+----------------------------------------------- Changes (by wjt): * cc: will@willthompson.co.uk (added) -- Ticket URL: http://hackage.haskell.org/trac/hackage/ticket/215#comment:6 Hackage http://haskell.org/cabal/ Hackage: Cabal and related projects
#215: Overhaul support for package's tests ----------------------------+----------------------------------------------- Reporter: duncan | Owner: Type: task | Status: new Priority: normal | Milestone: Cabal-1.8 Component: Cabal library | Version: Severity: normal | Resolution: Keywords: | Difficulty: project(> week) Ghcversion: 6.8.2 | Platform: ----------------------------+----------------------------------------------- Changes (by tibbe): * cc: johan.tibell@gmail.com (added) -- Ticket URL: http://hackage.haskell.org/trac/hackage/ticket/215#comment:7 Hackage http://haskell.org/cabal/ Hackage: Cabal and related projects
#215: Overhaul support for package's tests ----------------------------+----------------------------------------------- Reporter: duncan | Owner: Type: task | Status: new Priority: normal | Milestone: Cabal-1.8 Component: Cabal library | Version: Severity: normal | Resolution: Keywords: | Difficulty: project(> week) Ghcversion: 6.8.2 | Platform: ----------------------------+----------------------------------------------- Changes (by peteg): * cc: peteg42@gmail.com (added) -- Ticket URL: http://hackage.haskell.org/trac/hackage/ticket/215#comment:8 Hackage http://haskell.org/cabal/ Hackage: Cabal and related projects
#215: Overhaul support for package's tests ----------------------------+----------------------------------------------- Reporter: duncan | Owner: Type: task | Status: new Priority: normal | Milestone: Cabal-1.8 Component: Cabal library | Version: Severity: normal | Resolution: Keywords: | Difficulty: project(> week) Ghcversion: 6.8.2 | Platform: ----------------------------+----------------------------------------------- Comment (by igloo): See also http://hackage.haskell.org/trac/ghc/ticket/2608 -- Ticket URL: http://hackage.haskell.org/trac/hackage/ticket/215#comment:9 Hackage http://haskell.org/cabal/ Hackage: Cabal and related projects
#215: Overhaul support for package's tests ----------------------------+----------------------------------------------- Reporter: duncan | Owner: Type: task | Status: new Priority: normal | Milestone: Cabal-1.8 Component: Cabal library | Version: Severity: normal | Resolution: Keywords: | Difficulty: project(> week) Ghcversion: 6.8.2 | Platform: ----------------------------+----------------------------------------------- Changes (by bos): * cc: bos@serpentine.com (added) -- Ticket URL: http://hackage.haskell.org/trac/hackage/ticket/215#comment:10 Hackage http://haskell.org/cabal/ Hackage: Cabal and related projects
#215: Overhaul support for package's tests ------------------------------+--------------------------------------------- Reporter: duncan | Owner: Type: task | Status: new Priority: normal | Milestone: Cabal-1.8 Component: Cabal library | Version: Severity: normal | Keywords: Difficulty: project(> week) | Ghcversion: 6.8.2 Platform: | ------------------------------+--------------------------------------------- Changes (by michal.palka): * cc: michal.palka@… (added) -- Ticket URL: http://hackage.haskell.org/trac/hackage/ticket/215#comment:11 Hackage http://haskell.org/cabal/ Hackage: Cabal and related projects
participants (1)
-
Hackage