Donald Bruce Stewart wrote:
Following recent discussion about a cross-implementation performance benchmark suite, based on nofib, I've gone and combined nofib with the great language shootout programs, and rewritten the build system to support cross implementation measurements.
Great work! ..but I wonder if the shootout is really the kind of code that is ideal for compiler benchmark. Typically (at least based on what I've seen of the submissions) they tend to be fairly heavily tuned, using optimizations that are a) obfuscating the code and b) tuned specifically for GHC. (Another potential issue that follows from this is how to resolve a modification to a benchmark that makes one compiler faster at the expense of another.) Wouldn't it be better to benchmark a more idiomatically correct codebase? -k