--- Sebastian Sylvan
I don't think his reaction is reasonable. Especially not if he indeed is claiming that the cases where C is favoured are unintentional. The fact that he seems to take suggestions for improvements as a personal insult hardly helps his case (that the benchmarks aren't intentionally geared to favour C).
Daniel's benchmarks are interesting, and instead of a paranoid and rude response from the people responsible, I would rather have liked to see if the results are
Let's not descend into needless ad-hominem attacks. I certainly cannot claim that the shootout is perfect, or that we do everything right. But Isaac has been a long-time champion of correcting many of the problems in the shootout that are largely the result of my ineptitude, and I think comments that imply a lack of rigor or desire for accuracy are more rightly pointed at me. the
same on the systems used in the shootout.
I think I answered this in another e-mail. I do see similar results on the shootout machine. Out of a desire for inclusiveness, we arbitrarily kept lower values of N so that certain scripting languages (mainly Ruby and Python) would not self-destruct when attempting to process larger values of N. Furthermore, I had arbitrarily restricted the timeout to 600 seconds to avoid running the shootout for weeks due to poor performance in some implementations. However, now that the main shootout tests are stabilized, it's not such a big deal to extend the timeout (as we have done for spectralnorm and others), and I think it would be good to do so for the Ackermann test. Thanks, -Brent