Re: [Haskell-cafe] Shootout favouring C

On 1/16/06, Isaac Gouy wrote:

...

...

Shootout favouring C On 1/16/06, Daniel Fischer wrote: Is it only my machime, or can you confirm that for the Ackermann benchmark, it's very good for C that they chose 9 and not a larger value?

...

Sebastian Sylvan wrote: This is interesting. Hopefully it's not intentional,

Pardon my rudeness but this really is getting a bit much!

Please keep to the true spirit of fictional crime writing and provide a motive for these evil characters who will stop at nothing to make Haskell seem some worse than C.

I think you're being a bit paranoid here, nobody's claiming that there's some hidden agenda, just that there are improvements that could be made to increase accuracy and fairness. Surely you can see how a benchmark where the top three entries all have the same reported time would be better if run for a longer period of time? The larger problem sets you run your benchmarks on, the more accurate the results will be. In this particular case, the results could even be substantially different (if Daniel's results were to carry over the the systems used in the shootout). Do you not agree that if the results of a benchmark significantly changes when increasing the problem set slightly, it's better to run at a larger problem set? I really don't understand why you get so upset when someone points out improvements that could be made to the benchmarks. Isn't it better to get more accurate (and fair) results? This type of responses from you will certainly not help if you're concerned that people think there is a deliberate effort to make C look good. Why would you be against improving the accuracy of the benchmarks in cases where that would make C look worse? Ideally every benchmark should be run at a problem size large enough so that the rankings of the entries don't change substantially when increasing it (this size could be found automatically). That's just plain old common sense for benchmarking. Nothing to take personally and get upset about. In practice this may not be feasible for all of the benchmarks (I assume the shootout machines have better things to do than to run benchmarks 24/7) so it may be best to have a ceiling of a few seconds or so for the fastest benchmark. /S -- Sebastian Sylvan +46(0)736-818655 UIN: 44640862