more automatic tests for containers

Especially to Johan and Milan, I'm thinking to introduce more automatic tests for the containers library. Let's take Data.Map as example: 1) doctest Some haddoc documents include the following style. -- > Data.Map.null (empty) == True If we modify it like -- >>> Data.Map.null (empty) -- True we can use doctest. And we can remove corresponding unit test from test/*-properties.hs. Note that I'm not sure how to specify errors with doctest at this moment. 2) test-framework-th If we replace "test_" with "case_", we can use test-framework-th to generate the long list of test cases. We don't need to update it when we add "prop_" and "case_". We can specify the number of QuickCheck test as follows: runghc -i.. -DTESTING map-properties.hs --maximum-generated-tests=500 Since this is test stuff, we don't have to add doctest and test-framework-th to containers.cabal. What do you think? If you like this, I will work and send you pull request. --Kazu

On Wed, 21 Dec 2011, Kazu Yamamoto (????) wrote:
What do you think? If you like this, I will work and send you pull request.
I don't know what these frameworks imply on the installation process. I would like to be able to compile 'containers' with every Haskell 98 compiler. If the test frameworks are itself basic Haskell and the tests are, too, this would be even better.

I don't know what these frameworks imply on the installation process. I would like to be able to compile 'containers' with every Haskell 98 compiler. If the test frameworks are itself basic Haskell and the tests are, too, this would be even better.
I think my proposal does not change basic aspect: - containers itself can be compiled with Haskell 98 compiler - tests already use test-framework which is not a part of Haskell 98 compiler If you want to compile tests with Haskell 98 compiler, it's another topic. Please propose it in another thread. --Kazu

On 21 December 2011 05:31, Kazu Yamamoto
Since this is test stuff, we don't have to add doctest and test-framework-th to containers.cabal.
You could add them to the build-depends of a test-suite section: http://haskell.org/cabal/users-guide/#test-suites Bas

Hi, Am Mittwoch, den 21.12.2011, 13:31 +0900 schrieb Kazu Yamamoto:
Especially to Johan and Milan,
I'm thinking to introduce more automatic tests for the containers library. Let's take Data.Map as example:
1) doctest
Some haddoc documents include the following style.
-- > Data.Map.null (empty) == True
If we modify it like
-- >>> Data.Map.null (empty) -- True
we can use doctest. And we can remove corresponding unit test from test/*-properties.hs.
I’m not sure what the benefit is here. containers is a “high profile” library which sees lots of effort. Introducing a new testing library (doctest) just to avoid having to add those test cases to the test suite does not seem to gain much, but takes away some flexibility while writing good documentation. Maybe the “==” form is more readable to some users?
2) test-framework-th
If we replace "test_" with "case_", we can use test-framework-th to generate the long list of test cases. We don't need to update it when we add "prop_" and "case_".
We can specify the number of QuickCheck test as follows:
runghc -i.. -DTESTING map-properties.hs --maximum-generated-tests=500
Since this is test stuff, we don't have to add doctest and test-framework-th to containers.cabal.
What do you think? If you like this, I will work and send you pull request.
For the same reason as above, I don’t think it is worth it. Maintaining the lists of cases is not nice, but the amount of work is not high in relation to the efforts going into the library, so the gain is litte. The downside is that the test suite cannot be run on architectures where Template Haskell is not supported. Given that containers does some bitshifting that can exhibit architecture dependent bugs, this could put us in the bad situation that the tests cannot easily be run on those machines that need the tests the most. If you would like to work on the test suites, maybe you can identify parts of the code that are under-tested and come up with test cases? Greetings, Joachim -- Joachim "nomeata" Breitner mail@joachim-breitner.de | nomeata@debian.org | GPG: 0x4743206C xmpp: nomeata@joachim-breitner.de | http://www.joachim-breitner.de/

On Wed, Dec 21, 2011 at 15:20, Joachim Breitner
Hi,
Am Mittwoch, den 21.12.2011, 13:31 +0900 schrieb Kazu Yamamoto:
Especially to Johan and Milan,
I'm thinking to introduce more automatic tests for the containers library. Let's take Data.Map as example:
1) doctest
Some haddoc documents include the following style.
-- > Data.Map.null (empty) == True
If we modify it like
-- >>> Data.Map.null (empty) -- True
we can use doctest. And we can remove corresponding unit test from test/*-properties.hs.
I’m not sure what the benefit is here. containers is a “high profile” library which sees lots of effort. Introducing a new testing library (doctest) just to avoid having to add those test cases to the test suite does not seem to gain much, but takes away some flexibility while writing good documentation. Maybe the “==” form is more readable to some users?
Consistent unit tests and documentation seems like a nice benefit to me (don't forget about modifying and deleting unit tests). However, I agree that the structure imposed by doctest might lead to less than tasteful documentation. -- Love in Jesus Christ, John Alfred Nathanael Chee http://www.biblegateway.com/ http://web.cecs.pdx.edu/~chee/

Consistent unit tests and documentation seems like a nice benefit to me (don't forget about modifying and deleting unit tests). However, I agree that the structure imposed by doctest might lead to less than tasteful documentation.
I agree. I should implement document test which evaluates examples of Data.Map (for instance) directly: ---- insert 5 'x' (fromList [(5,'a'), (3,'b')]) == fromList [(3, 'b'), (5, 'x')] insert 7 'x' (fromList [(5,'a'), (3,'b')]) == fromList [(3, 'b'), (5, 'a'), (7, 'x')] insert 5 'x' empty == singleton 5 'x' ---- And sees if they return True. --Kazu

Hello,
Consistent unit tests and documentation seems like a nice benefit to me (don't forget about modifying and deleting unit tests). However, I agree that the structure imposed by doctest might lead to less than tasteful documentation.
The following hack to doctest enables that empty example results are treated as True. https://github.com/kazu-yamamoto/doctest-haskell/commit/ab124abf6791870fd8f9... So, if we convert -- > Data.Map.null (empty) == True to -- >>> Data.Map.null (empty) == True the modified doctest can handle this example as test case. If people like this approach, I will send a pull request to the author of doctest and convert the examples of the containers library. --Kazu

On 12/22/11 9:35 PM, John Alfred Nathanael Chee wrote:
On Wed, Dec 21, 2011 at 15:20, Joachim Breitner
wrote: I’m not sure what the benefit is here. containers is a “high profile” library which sees lots of effort. Introducing a new testing library (doctest) just to avoid having to add those test cases to the test suite does not seem to gain much, but takes away some flexibility while writing good documentation. Maybe the “==” form is more readable to some users?
Consistent unit tests and documentation seems like a nice benefit to me (don't forget about modifying and deleting unit tests). However, I agree that the structure imposed by doctest might lead to less than tasteful documentation.
I definitely prefer the "==" form for stating laws the interface follows. Doctest came from the Python tradition, which is a very imperative outlook on life. I wonder whether doctest could be modified to better match the programming, documentation, and testing style of Haskell. In particular, I'd expect it to recognize "=="-style statements as things to test. The only real issue is whether those statements should be considered unit tests or properties (testable by QuickCheck and SmallCheck), since those two categories have very different methods of being verified. -- Live well, ~wren

I definitely prefer the "==" form for stating laws the interface follows.
Doctest came from the Python tradition, which is a very imperative outlook on life. I wonder whether doctest could be modified to better match the programming, documentation, and testing style of Haskell. In particular, I'd expect it to recognize "=="-style statements as things to test.
Please read: http://www.haskell.org/pipermail/libraries/2011-December/017382.html
The only real issue is whether those statements should be considered unit tests or properties (testable by QuickCheck and SmallCheck), since those two categories have very different methods of being verified.
I agree. P.S. I have extended test-framework-th to handle doctest (HUnit) also. I will probably release this in this week. --Kazu

Doctest came from the Python tradition, which is a very imperative outlook on life. I wonder whether doctest could be modified to better match the programming, documentation, and testing style of Haskell. In particular, I'd expect it to recognize "=="-style statements as things to test. The only real issue is whether those statements should be considered unit tests or properties (testable by QuickCheck and SmallCheck), since those two categories have very different methods of being verified.
Isn't a unit test just a property with 0 free variable? If so, equality statements can always be handled by Quick/SmallCheck. Cheers, JP.
participants (7)
-
Bas van Dijk
-
Henning Thielemann
-
Jean-Philippe Bernardy
-
Joachim Breitner
-
John Alfred Nathanael Chee
-
Kazu Yamamoto
-
wren ng thornton