[Haskell-cafe] HUnit false-positive stumper

6 Jun 2011

      While working on a project of mine recently I realized that a particular HUnit test should have been failing.  After paring things down, I came up with this shocker:

test_perhaps.hs:
...
module Main where
import Test.HUnit
main = runTestTT $ TestList [ True  ~=? True
                            , False ~=? True
                            , TestCase $ assertEqual "both true" True True
                            , TestCase $ assertEqual "false true" False True
                            ]
$ cabal-dev install
...
$ ./cabal-dev/bin/test_perhaps.hs
### Failure in: 3
false true
expected: False
  but got: True
Cases: 4  Tried: 4  Errors: 0  Failures: 1

The shock here is that there was only one failure, whereas the "False ~=? True" should have failed.
Environment: MacOS 10.5.8  (Leopard)
              GHC 6.12.3
              cabal-dev 0.7.4.1
              Cabal 1.10.1.0
              HUnit 1.2.2.3
I even tried removing the dist and cabal-dev directories and reattempting but got the same results.

I tried the same test in a different environment (Linux PC w/GHC 7.0.2, all other elements the same) and got correct results:

$ ./cabal-dev/bin/test_perhaps
### Failure in: 1
expected: False
  but got: True
### Failure in: 3
false true
expected: False
  but got: True
Cases: 4  Tried: 4  Errors: 0  Failures: 2
$

I realize it's very difficult to debug someone's setup remotely, but I have to confess I'm really stumped at this point.  The input program is pretty straightforward, and I *have* gotten test failures in the past on the Mac.  I did do a "cabal update" the other day after being informed that my hackage file was getting old, but HUnit hasn't changed in months.

I even tried unpacking HUnit and reducing the (non-)failing test case by using the definitions of the elements:
...
module Main where
import Control.Monad (unless)
import Test.HUnit
main = runTestTT $ TestList [ True  ~=? True
                            , False ~=? True
                            , TestCase $ assertEqual "both true" True True
                            , TestCase $ assertEqual "false true" False True
                            , TestCase $ assertEqual "fa" False True
                            , TestCase $ assertEqual "f" False True
                            , TestCase $ (False @?= True)
                            , TestCase $ unless (False == True) (assertFailure "f")
                            ]
The results are very strange:

$ ./cabal-dev/bin/test_perhaps
### Failure in: 3
false true
expected: False
  but got: True
### Failure in: 4
fa
expected: False
  but got: True
### Failure in: 7
f
Cases: 8  Tried: 8  Errors: 0  Failures: 3
$

The assertEqual form doesn't fail as it should if the label is a single character (test 5), but in its fully expanded form (test 7) it will fail.  Huh?!

At this point I'm thoroughly confused.  I'm using 6.12.3 on the Mac because I don't have the newer MacOS release for which there's a Haskell Platform release and I haven't wanted to build GHC by hand.  I wouldn't expect 6.12.3 to have issues like this, but I wouldn't expect issues like this anywhere.  If anyone has any suggestions (other than upgrading to GHC 7.x) I'll be most appreciative.

-- 
-KQ