Re: [Haskell-cafe] Shootout favouring C

16 Jan 2006

      Am Montag, 16. Januar 2006 18:56 schrieb Isaac Gouy:
...
...
...
Shootout favouring C
On 1/16/06, Daniel Fischer wrote:
Is it only my machime, or can you confirm that for
the Ackermann benchmark, it's very good for C that
they chose
...
...
9 and not a larger value?
Sebastian Sylvan  wrote:
This is interesting. Hopefully it's not intentional,
Although it'd give a great thread, just think what you'd have to do.
You'd have to write _good_ programmes in many languages and then try a lot of 
different inputs...

Probably, the Ack(3,9) benchmark was just taken over from the times when it 
took serious time to calculate.

However, I'm curious. Is the behaviour I reported peculiar to my flavour of 
linux, or does the same trend show on other machines?
Would you be so kind to test and report?
...
Pardon my rudeness but this really is getting a bit
much!
Please keep to the true spirit of fictional crime
writing and provide a motive for these evil characters
who will stop at nothing to make Haskell seem some
worse than C.
Jealousy?
...
...
but it's quite obvious that for benchmarks where the fastest
time is only a few fractions of a second, languages with more
complex runtime systems will be unfairly slow due to the
startup cost.
Sebastian perhaps you'd like to provide something more
substantive than "quite obvious".
Well, if initializing the run-time environment takes as long as for Java
(boy, I just timed Hello, World: 0.34s!!!!), no matter how fast the actual 
work is being done, you can't catch sufficiently fast languages with 
lightweight RTEs in those cases -- and I dare say Java does suffer from that 
even more than GHC.
...
...
There is already a startup benchmark in there
Yes and if we make the huge assumption that it means
anything at all, then we are being unfair to Haskell
by 0.002s on every test - we only show measurements to
0.01s!
And you take the average of how many runs?
From my experience, even if I run nothing else, for tasks which take around 
0.1s, timings vary by up to 0.03s (and if the system starts some real work 
while the task is underway, it can be more).
So I don't consider timings of such short tasks very reliable and second 
Sebastians suggestion that all benchmarks should take at least -- well, I 
don't know how much -- 0.5s, 1s for the fastest contenders.
And that's not because apparently -- it might be violently different on other 
systems -- Haskell would gain ground over C if the Ackermann benchmark would 
be increased (though I'm happy if it does).
And though I've no reason to suppose it would help Haskell, for the same 
reasons, I'd like the fannkuch benchmark changed to Pfannkuchen(10).
...
...
In other words I'd prefer if all benchmarks are
reconfigured to target an execution time of at least a few
seconds for the fastest benchmarks.
We run the Haskell regex-dna programs for 2500s -
isn't that long enough?
Phew, 'tis.
The old song, not every language is good at everything...
...
Let me join Simon Marlow in congratulating those who
are using the Shootout to advertise what Haskell can
do, by the straightforward approach of contributing
faster, smaller, more elegant programs.
Agreed. Unfortunately, often the desire for speed wrecks elegance.
...
best wishes, Isaac
Cheers,
Daniel