G'day all. On Mon, Jul 29, 2002 at 10:23:05AM +0100, Simon Marlow wrote:
Many of those programs can be written differently to improve performance.
To be fair, Doug admits this as well as a lot more: http://www.bagley.org/~doug/shootout/method.shtml#flaws Despite these flaws, I did notice that ghc is right in the middle in his CRAPS score system (which is really interesting; all due respect to the GHC guys, but I expected it to be lower <g>). I also noticed that the majority of cases where Haskell does significantly worse than average are "same way" tests, designed to test the performance of various constructs (e.g. array access, dictionary lookup) as opposed to "same thing" tests, designed to test native idioms. In the end, though, benchmarks ignore one of the most important rules of software performance: "throughput" (i.e. the amount of processing that your system can do just prior to being overloaded) is almost never the most important consideration. Other considerations such as flexibility, robustness, responsiveness and scalability are almost always more important. I've thought for a while that what we need is more benchmarks like pseudoknot: Real tasks which real people want to do. Computing Ackermann's function is all well and good, but when's the last time you actually needed to compute it in a real program? Off the top of my head, some "real" tasks which could be benchmarked include: - MPEG video compression. - Code scheduling/register allocation for a CPU like the MIPS/Alpha or even the IA64. - Fluid simulation. - Solving chess problems. Cheers, Andrew Bromage