Malcolm Wallace wrote:
For instance, the shootout often requires that a task be carried out N times, to make the timings large enough to measure. In all the naive Haskell implementations of these tasks, Haskell wins by a mile. Why? Because the language quite reasonably says that if you multiply a matrix by itself N times, but only use the result of the last multiplication, well it is jolly well not going to bother computing the first (N-1) identical multiplications - what a waste of time!
So is it fair to compare the default lazy Haskell solution with all the eager solutions out there that laboriously do all this unnecessary work? Apparently not, so we have gone to all kinds of trouble to slow the Haskell solution down, make it over-strict, do the work N times, and thereby have a "fair" performance test. Huh.
Well, the shootout appears to have two types of tests. Each individual test is labeled as either being implemented in the "Same Way" or doing the "Same Thing". I agree that the "Same Way" test are usually too synthetic and too geared toward measuring artifacts of imperative programming which aren't appropriate in Haskell. But there are also the "Same Thing" tests that are free to be coded in any style at all. And of the "Same Thing" tests, the largest slowdown in the GHC programs is caused by lazy I/O (I think using better I/O routines would fix "Reverse a File" and "Statistical Moments"). To get GHC to come out on top of the CRAPS Scorecard, we need to emphasize the "Same Thing" tests and downplay the "Same Way" tests as well as properly weighting lines of code vs. memory consumption and speed. Here's one way to do it... http://makeashorterlink.com/?O5BD42089 What we probably need to do is create some new tests which aren't as phony and convince the powers-that-be to drop some of the synthetic tests or convert them to "Same Thing" tests (I think the "Sum a Column of Integers" and "Spell Checker" would be good candidates to convert). Just for reference, here's a list of the "Same Thing" tests: Simple TCP/IP Server Matrix Multiplication Statistical Moments Process Instantiation Reverse A File Ring of Messages Count Lines/Words/Chars Word Frequency Greg Buchholz