[Fwd: Re: Computer Language Shootout]
Forwarding on behalf of Andrzej Jaworski <himself@poczta.nom.pl>: -------- Original Message -------- From: Andrzej Jaworski <himself@poczta.nom.pl> Dear fellows, It is ironic that just after SPJ disclosed Comments from Brent Fulgham on Haskell and the shootout the situation has radically changed for the worse. Without knowing that I committed a blunder referring to the damn benchmark http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all together with multicore support arguments when trying to convert a prominent OCaml programmer to Haskell. Now they know more of us:-) What a language it is that jumps 30% up and down on benchmark while other languages gracefully stay in line? Can any of you explain the reason for this disgraceful degradation in Computer Language Shootout? Clean has also declined in these benchmarks but not that much as Haskell. According to John van Groningen Clean's binary-trees program in the previous shootout version used lazy data structure which resulted in lower memory usage and much faster execution. That was removed by the maintainer of the shootout and replaced by a much slower one using strict data structure. I fear that if laziness accounted for previous good scores of GHC then algorithms where laziness is downplayed must be responsible for that anecdotal opinion that Haskell can be extremely slow. For example: on Royal Road Problem in genetic algorithms http://www.dcs.ed.ac.uk/home/stg/sfpw/book/Garmendia-Doval/cameraready.ps Haskell was found to be on average over 500 times slower than SML implementation! If such extreme variations in performance are inherent for Haskell then the support for multicore rather than boosting its relative performance (against e.g. one-core-bound OCaml) may merely amplify Haskell unevenness to the point where programming becomes more of an art than a science like it is in Prolog. Perhaps making a collective effort towards benchmarking Haskell programs and analyzing the results in some methodic way could prove helpful? Regards, Andrzej Jaworski
Where are SPJs disclosed comments from Brent Fulgham? On Jan 25, 2007, at 8:55 AM, Simon Marlow wrote:
Forwarding on behalf of Andrzej Jaworski <himself@poczta.nom.pl>:
-------- Original Message -------- From: Andrzej Jaworski <himself@poczta.nom.pl>
Dear fellows,
It is ironic that just after SPJ disclosed Comments from Brent Fulgham on Haskell and the shootout the situation has radically changed for the worse.
http://www.mail-archive.com/haskell@haskell.org/msg18863.html (It was Simon Marlow actually, but we are joined at the hip so it hardly matters) | -----Original Message----- | From: haskell-bounces@haskell.org [mailto:haskell-bounces@haskell.org] On Behalf Of Joel Reymont | Sent: 25 January 2007 09:01 | To: Simon Marlow | Cc: haskell@haskell.org | Subject: Re: [Haskell] [Fwd: Re: Computer Language Shootout] | | Where are SPJs disclosed comments from Brent Fulgham? | | On Jan 25, 2007, at 8:55 AM, Simon Marlow wrote: | | > Forwarding on behalf of Andrzej Jaworski <himself@poczta.nom.pl>: | > | > -------- Original Message -------- | > From: Andrzej Jaworski <himself@poczta.nom.pl> | > | > Dear fellows, | > | > It is ironic that just after SPJ disclosed Comments from Brent | > Fulgham on | > Haskell and the shootout the situation has radically changed for | > the worse.
(Simon and Andy, if you guys got this twice, sorry about the double mail) On 25 Jan 2007, at 09:55, Simon Marlow wrote:
Forwarding on behalf of Andrzej Jaworski <himself@poczta.nom.pl>:
-------- Original Message -------- From: Andrzej Jaworski <himself@poczta.nom.pl>
<snip>
Perhaps making a collective effort towards benchmarking Haskell programs and analyzing the results in some methodic way could prove helpful?
In which way? Using hardware performance counter metrics on actual hardware? Or determining which bottlenecks occur with Haskell programs in binary form (in terms of data locality and the lot)? Is there a 'Haskell-program benchmark suite' out there? greetings, Kenneth -- Statistics are like a bikini. What they reveal is suggestive, but what they conceal is vital (Aaron Levenstein) Kenneth Hoste ELIS - Ghent University kenneth.hoste@elis.ugent.be http://www.elis.ugent.be/~kehoste
On Thu, Jan 25, 2007 at 08:55:37AM +0000, Simon Marlow wrote:
Clean has also declined in these benchmarks but not that much as Haskell. According to John van Groningen Clean's binary-trees program in the previous shootout version used lazy data structure which resulted in lower memory usage and much faster execution. That was removed by the maintainer of the shootout and replaced by a much slower one using strict data structure.
Why was this done? I notice that a lot of people expouse the 'strictness is good, lazy is bad' methodology of optimization. We really should be careful about that, naive additions of seqs (or even worse, deepSeqs!) can kill algorithmic performance of code and create bad space leaks. Lazy is good, it is much better to never compute something at all then compute it just a bit faster. Haskell is lazy by default and users need to start thinking in terms of lazyness by default to fully take advantage of haskell's goodness. Not that careful manipulation of strictness isn't key to good performance, but the 'shotgun' approach of just adding 'deepSeqs' and 'seq's everywhere (without benchmarking to see if it actually helps) should be avoided and certainly not advocated to new users. Although it is especially hard to generalize optimization rules for haskell, I think the closest one can to a rule of thumb is "make operations on integral or basic types as strict as possible, make everything else as lazy as possible.". at least, that is as far as one should go without benchmarks or some good reasoning to manipulate strictness further. John -- John Meacham - ⑆repetae.net⑆john⑈
On 1/25/07, John Meacham <john@repetae.net> wrote:
On Thu, Jan 25, 2007 at 08:55:37AM +0000, Simon Marlow wrote:
Clean has also declined in these benchmarks but not that much as Haskell. According to John van Groningen Clean's binary-trees program in the previous shootout version used lazy data structure which resulted in lower memory usage and much faster execution. That was removed by the maintainer of the shootout and replaced by a much slower one using strict data structure.
Why was this done?
Careful about your attributions! The passage you quoted was from Andrzej Jaworski's message that Simon forwarded, and was not written by Simon. (I don't know the answer to your actual question, but maybe answers to it should go to haskell-cafe?) Cheers, Kirsten -- Kirsten Chevalier* chevalier@alum.wellesley.edu *Often in error, never in doubt "Getting an education was a bit like a communicable sexual disease. It made you unsuitable for a lot of jobs, and then you had the urge to pass it on." -- Terry Pratchett
* John Meacham:
Clean has also declined in these benchmarks but not that much as Haskell. According to John van Groningen Clean's binary-trees program in the previous shootout version used lazy data structure which resulted in lower memory usage and much faster execution. That was removed by the maintainer of the shootout and replaced by a much slower one using strict data structure.
Why was this done?
I suppose the itent of the binary-trees benchmark is to measure allocation performance in the presence of a fairly large (well, not in today's terms) data structure. Using laziness to prevent that data structure from being built (or use additional sharing) kind of defeats the purpose of the benchmark. Note that these are microbenchmarks, not real applications. Imposing such rules makes sense.
fw:
* John Meacham:
Clean has also declined in these benchmarks but not that much as Haskell. According to John van Groningen Clean's binary-trees program in the previous shootout version used lazy data structure which resulted in lower memory usage and much faster execution. That was removed by the maintainer of the shootout and replaced by a much slower one using strict data structure.
Why was this done?
I suppose the itent of the binary-trees benchmark is to measure allocation performance in the presence of a fairly large (well, not in today's terms) data structure. Using laziness to prevent that data structure from being built (or use additional sharing) kind of defeats the purpose of the benchmark.
Note that these are microbenchmarks, not real applications. Imposing such rules makes sense.
Agreed. I've submitted a strict variant that should allocate similarly to OCaml. I'd suggest stating this requirement for strict allocation in the spec. Regards, Don
It sounds reasonable. However knowledge of how program performs in micro-steps does not add up, so the benchmarks may wet up appetite for lunch that does not come. I have pointed into such example - an astonishing and unexplained underperformance of Haskell with all the profiling information at hand. I guess Haskell compilers are not particularly good at detecting specific properties of a program and hence with optimizing it. This however shows up with size so Donald's benchmarks cannot catch that out. For this reason, undiagnosed and untreated, Haskell has been abandoned for example in Algebraic Dynamic Programming, in spite of its unparallel expressive power and a lot of hope. In ILP/IFP and GP it failed too. Cheers, --Andrzej
On Mon, Feb 26, 2007 at 03:43:08AM +0100, Andrzej Jaworski wrote:
It sounds reasonable. However knowledge of how program performs in micro-steps does not add up, so the benchmarks may wet up appetite for lunch that does not come. I have pointed into such example - an astonishing and unexplained underperformance of Haskell with all the profiling information at hand.
I guess Haskell compilers are not particularly good at detecting specific properties of a program and hence with optimizing it. This however shows up with size so Donald's benchmarks cannot catch that out.
For this reason, undiagnosed and untreated, Haskell has been abandoned for example in Algebraic Dynamic Programming, in spite of its unparallel expressive power and a lot of hope. In ILP/IFP and GP it failed too.
C, thirty years ago: (disclaimer, I'm 16) * Very much slower than assembly * Very much easier to use than assembly * Very easy to interface with assembly So everyone used C with assembler inner loops, no big deal. Haskell, now: * Very much slower than C * Very much easier to use than C * Very easy to interface with C So I think we should do the same. It even shows in the Shootout - the programs that are simultaneously fastest and clearest are not pure Haskell, but delegate their innermost loops to tuned C libraries (FPS and GMP). Stefan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sun, Feb 25, 2007 at 06:57:29PM -0800, Stefan O'Rear wrote:
Haskell, now: * Very easy to interface with C
I might buy "as easy as interfacing a GCed language with a non-GCed language can reasonably be". When the GC abstraction leaks, it leaks angry wasps. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFF4k8+KTA17JAC/eYRApLXAKDDfGxlcHubvSUYjaeDxxIq+xzj8QCeNmsp If52gkdoa5ufeQZclnNbTKo= =s+B5 -----END PGP SIGNATURE-----
On Sun, 2007-02-25 at 18:57 -0800, Stefan O'Rear wrote:
Haskell, now: * Very much slower than C * Very much easier to use than C * Very easy to interface with C
So I think we should do the same. It even shows in the Shootout - the programs that are simultaneously fastest and clearest are not pure Haskell, but delegate their innermost loops to tuned C libraries (FPS and GMP).
I should note that FPS is almost completely Haskell code, not C. We use C for things like memcmp, memcpy, memset, reverse_copy, intersperse, maximum, minimum and count. Certainly some of the innards are low level style Haskell, though not the kind that could be replicated in C because we use high level transformations to fuse loop bodies together and wrap them in high performance, low level loop code. This is not the style where we just wrap well tuned C code, this is a style where we generate high performance low level code from a high level spec. This relies on GHC's excellent and programmable optimiser. It's wrong to say that the shootout improvements were only down to improved libraries. The performance of ByteString code improved very significantly between GHC 6.4 and 6.6 and a large part of that was down to optimiser improvements (not just the ForeignPtr rep change). Duncan
duncan.coutts:
On Sun, 2007-02-25 at 18:57 -0800, Stefan O'Rear wrote:
Haskell, now: * Very much slower than C * Very much easier to use than C * Very easy to interface with C
So I think we should do the same. It even shows in the Shootout - the programs that are simultaneously fastest and clearest are not pure Haskell, but delegate their innermost loops to tuned C libraries (FPS and GMP).
I should note that FPS is almost completely Haskell code, not C. We use C for things like memcmp, memcpy, memset, reverse_copy, intersperse, maximum, minimum and count.
Certainly some of the innards are low level style Haskell, though not the kind that could be replicated in C because we use high level transformations to fuse loop bodies together and wrap them in high performance, low level loop code.
This is not the style where we just wrap well tuned C code, this is a style where we generate high performance low level code from a high level spec. This relies on GHC's excellent and programmable optimiser.
It's wrong to say that the shootout improvements were only down to improved libraries. The performance of ByteString code improved very significantly between GHC 6.4 and 6.6 and a large part of that was down to optimiser improvements (not just the ForeignPtr rep change).
And just to point out that the optimiser is even better in GHC Head. Thanks Simon! Here's today's run on an amd64, with GHC 6.6 versus GHC head, http://www.cse.unsw.edu.au/~dons/nobench/x86_64/results.html For microbenchmarks, the head is 24% faster, across the whole suite 8%. Note the bytestring-based program, sum-col, got 40% faster purely due to improvements in the optimiser! -- Don
I am new to haskell, but I find your assertions surprising, given that from my experience the really performance critical code is little, and the reset can be even interpreted. As far as I know C/C++ or similar are not really that advanced with respect to whole program optimization (not much more than inlining). I had the impression that haskell, until the shootout push, was not good at optimizing/had not optimized libraries for some common computational kernels, but now is in a much better shape (for ghc), and with Don is doing, hopefully it will stay so. Can you corroborate a little more your points? cheers Fawzi On Feb 26, 2007, at 3:43 AM, Andrzej Jaworski wrote:
It sounds reasonable. However knowledge of how program performs in micro-steps does not add up, so the benchmarks may wet up appetite for lunch that does not come. I have pointed into such example - an astonishing and unexplained underperformance of Haskell with all the profiling information at hand.
I guess Haskell compilers are not particularly good at detecting specific properties of a program and hence with optimizing it. This however shows up with size so Donald's benchmarks cannot catch that out.
For this reason, undiagnosed and untreated, Haskell has been abandoned for example in Algebraic Dynamic Programming, in spite of its unparallel expressive power and a lot of hope. In ILP/IFP and GP it failed too.
Cheers, --Andrzej
_______________________________________________ Haskell mailing list Haskell@haskell.org http://www.haskell.org/mailman/listinfo/haskell
Hello Fawzi, Monday, February 26, 2007, 3:44:09 PM, you wrote:
I am new to haskell, but I find your assertions surprising, given that from my experience the really performance critical code is little, and the reset can be even interpreted. As far as I know C/C++ or similar are not really that advanced with respect to whole program optimization (not much more than inlining).
yes, Haskell provides better opportunities of higher-level program optimizations and i guess that ghc better in this aspect than traditional C compilers. but C programmers just make such sort of optimizations manually. on the side of lower-level optimizations i guess that only jhc currently can compete with C compilers - only for low-level optimized Haskell code (that is harder to write that low-level optimized C code) and only because jhc generates C itself as intermediate code
I had the impression that haskell, until the shootout push, was not good at optimizing/had not optimized libraries for some common computational kernels, but now is in a much better shape (for ghc), and with Don is doing, hopefully it will stay so.
naive youth :) first, most of shootout entries depend on libraries speed (multithreading, regexps), not speed of code generated by the compiler. second, ghc was not changed much, but programs was rewritten in the non-idiomatic way. you can find ideas of such rewriting in http://www.cse.unsw.edu.au/~dons/papers/fusion.pdf http://www.cse.unsw.edu.au/~chak/papers/afp-arrays.ps.gz -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
participants (14)
-
Andrzej Jaworski -
Bulat Ziganshin -
Daniel Franke -
dons＠cse.unsw.edu.au -
Duncan Coutts -
Fawzi Mohamed -
Florian Weimer -
Joel Reymont -
John Meacham -
Kenneth Hoste -
Kirsten Chevalier -
Simon Marlow -
Simon Peyton-Jones -
Stefan O'Rear