attoparsec double precision, quickCheck and aeson

The double parser provided by Data.Attoparsec.ByteString.Char8 looses precision around the 13-15th decimal place (http://hackage.haskell.org/packages/archive/attoparsec/0.10.2.0/doc/html/Dat...). Unfortunately this reeks havoc with my attempts to write a quickCheck test that validates print-read equivalence for a program that uses Aeson. I have tried compensating for this round-off error in my quickCheck generator using a function like this: roundDouble :: Double -> Double roundDouble d = let Right v = A8.parseOnly A8.double (C.pack (show d)) in v which helps in many cases, but for some the parsing seems bi-stable, alternating between two imprecise double values and causing the test to fail. I was wondering if anyone could suggest a better work-around for this problem, or explain why Attoparsec's double parser can't be isomorphic to haskell's. Thanks, Warren

On Tue, Jun 5, 2012 at 9:12 AM, Warren Harris
which helps in many cases, but for some the parsing seems bi-stable, alternating between two imprecise double values and causing the test to fail.
You want to perform your test as d1 - d2 < epsilon where epsilon is derived from the relative error you're willing to accept (e.g. 0.01% error.) You can't use an absolute epsilon because if you pick e.g. epsilon = 0.00001 but your input are also very small, you'll end up accepting a big relative error. Comparing floating points values for equality is asking for trouble.
I was wondering if anyone could suggest a better work-around for this problem, or explain why Attoparsec's double parser can't be isomorphic to haskell's. Thanks,
I think attoparsec uses a slightly less exact but much much faster double parser. I believe it's described in the attoparsec haddock docs somewhere. -- Johan

On Jun 5, 2012, at 9:38 AM, Johan Tibell wrote:
You want to perform your test as
d1 - d2 < epsilon
What's the best way to do this though, since aeson's Value type already provides instance Eq? I guess I can write my own suite of equality comparisons, but these aeson values are the leaves of an extensive grammar and it seems a shame to mimic everything that Eq does (without the benefits of deriving) just to compare doubles differently.

On Tue, Jun 5, 2012 at 9:51 AM, Warren Harris
On Jun 5, 2012, at 9:38 AM, Johan Tibell wrote:
You want to perform your test as
d1 - d2 < epsilon
What's the best way to do this though, since aeson's Value type already provides instance Eq? I guess I can write my own suite of equality comparisons, but these aeson values are the leaves of an extensive grammar and it seems a shame to mimic everything that Eq does (without the benefits of deriving) just to compare doubles differently.
I don't think applying == to something that contains floating point values at the leaves makes much sense. You want some approxEq function that uses approximate equality on floating point value or you want to equality function that ignores the floating point values. Probably not the answer you like, but I don't know how to define Eq in a robust way for types that include floating point values. -- Johan

On Jun 5, 2012, at 9:57 AM, Johan Tibell wrote:
I don't think applying == to something that contains floating point values at the leaves makes much sense. You want some approxEq function that uses approximate equality on floating point value or you want to equality function that ignores the floating point values. Probably not the answer you like, but I don't know how to define Eq in a robust way for types that include floating point values.
I buy that in general for comparing floats (those that result from arithmetic operations), but this is a case where attoparsec's parser is munging the value. I would like to have a law that says "parse . print == id" ... which is why this seems more like a bug than the usual floating point concerns. This law seems to hold for haskell's double parser: quickCheck (\d -> read (show d) == d)

On Tue, Jun 5, 2012 at 9:12 AM, Warren Harris
which helps in many cases, but for some the parsing seems bi-stable, alternating between two imprecise double values and causing the test to fail. I was wondering if anyone could suggest a better work-around for this problem, or explain why Attoparsec's double parser can't be isomorphic to haskell's.
If you need the full precision, use rational instead. The double parser is there because parsing floating point numbers is often a bottleneck, and double intentionally trades speed for precision.

On Tue, Jun 5, 2012 at 10:51 AM, Bryan O'Sullivan
If you need the full precision, use rational instead. The double parser is there because parsing floating point numbers is often a bottleneck, and double intentionally trades speed for precision.
Relevant code in https://github.com/bos/attoparsec/blob/master/Data/Attoparsec/ByteString/Cha... -- Johan

On Jun 5, 2012, at 10:51 AM, Bryan O'Sullivan wrote:
On Tue, Jun 5, 2012 at 9:12 AM, Warren Harris
wrote: which helps in many cases, but for some the parsing seems bi-stable, alternating between two imprecise double values and causing the test to fail. I was wondering if anyone could suggest a better work-around for this problem, or explain why Attoparsec's double parser can't be isomorphic to haskell's.
If you need the full precision, use rational instead. The double parser is there because parsing floating point numbers is often a bottleneck, and double intentionally trades speed for precision.
I'm actually using Aeson and the parser it provides, so I don't really have the option of using rational.
participants (3)
-
Bryan O'Sullivan
-
Johan Tibell
-
Warren Harris