On 05/02/13 23:48, Johan Tibell wrote:
I've now added the shootout programs that could be added without modifying the programs itself. I described why some programs weren't added in nofib/shootout/README.
For the curious, here's the change in these benchmarks from 7.0.4 to 7.6.2:
-------------------------------------------------------------------------------- Program Size Allocs Runtime Elapsed TotalMem -------------------------------------------------------------------------------- binary-trees +2.6% -0.6% -2.8% -2.8% -22.3% fannkuch-redux +1.4%+11514445. +0.2% +0.2% +0.0% n-body +3.8% +0.0% +4.4% +4.4% +0.0% pidigits +2.2% -6.9% -1.7% -1.2% -20.0% spectral-norm +2.1% -61.3% -54.8% -54.8% +0.0% -------------------------------------------------------------------------------- Min +1.4% -61.3% -54.8% -54.8% -22.3% Max +3.8%+11514445. +4.4% +4.4% +0.0% Geometric Mean +2.4% +737.6% -14.7% -14.6% -9.1%
This is slightly off topic, but I wanted to plant this thought in people's brains: we shouldn't place much significance in the average of a bunch of benchmarks (even the geometric mean), because it assumes that the benchmarks have a sensible distribution, and we have no reason to expect that to be the case. For example, in the results above, we wouldn't expect a 14.7% reduction in runtime to be seen in a typical program. Using the median might be slightly more useful, which here would be something around 0% for runtime, though still technically dodgy. When I get around to it I'll modify nofib-analyse to report medians instead of GMs. Cheers, Simon