On 29/01/07, Simon Peyton-Jones
We found these categories to be useful and robust, and I think they'd be useful for the new suite. In particular, the imaginary suite is useless for (say) choosing a compiler, but fantastic for exposing particular weak spots. But if the imaginary programs were mixed with the real ones, the whole thing would lose credibility.
One thing I'd like to see, but I have no idea if it would be practical, is an attempt to distinguish between "idiomatic" and "highly optimised" benchmarks. Given that optimising Haskell code is a complex problem - and often unfamiliar to programmers coming from other backgrounds - this would be useful for isolating how well a compiler does at dealing with code which is *not* written purely for speed. As I say, though, this may be impossible to achieve in practice (although something may be possible at the "imaginary program" level). Paul.