Comments from Brent Fulgham on Haskell and the shootout
Brent Fulgham has given me permission to share this message that he sent to us about his views on Haskell and the Great Computer Language Shootout. There's some nice advocacy material in here. I'd like to publicly thank Brent for all his work on the shootout - benchmarking 50 or so different language implementations and getting any kind of useful results at all is a significant feat. The shootout results have also influenced our direction in the Haskell community and GHC in particular. Here's the message, for your enjoyment... -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear Drs. Hudak and Peyton-Jones: (Feel free to forward these comments to anyone you think would enjoy them). A couple of years ago I revived the dormant "Great Computer Language Shootout" website because I wanted to create an up-to-date comparison of language implementations to use as ammunition against my nay- saying coworkers over the state of the art in Computer Science. Back in those days, I was mainly interested in the performance of Objective Caml and Lisp, but was also interested in a new (to me) language called Haskell . Haskell had (and still has) a beautiful, mathematical notation that was very easy to read, but it was such a poor performer that I despaired of ever getting to use it for anything but simple test programs. While it was conceptually satisfying, pragmatism seemed destined to keep it on the shelf for the foreseeable future. I wanted to write to inform you how shocked I was to see the great advances in performance in the Glorious Haskell Compiler over the last year or so. Of course, we have also benefited from some great contributions by the folks on the Haskell-Cafe mailing list. The thing I find most staggering about the compiler is that its quite brilliant at identifying wasted work. Many of the original shootout test cases were written to test the performance of simple scripting languages, and had grown a set of hacks over the years to keep the run times large enough to measure as processor speeds improved and language implementations became more sophisticated. In many cases, tests involved calculation or large values in loops, which would cause Perl and Python (and even C, Java, and C++) to chug merrily away burning cycles so that I could measure the timings in seconds, rather than microseconds. Then along came GHC, which began achieving enormous wins over the other languages because the lazy nature of the evaluation threw out the unneeded calculations, handing over the final useful result without breaking a sweat. Consequently, we had to revise many of the tests to force the programs to perform real work that could not be avoided, in some cases by forcing intermediate values to be printed in the solutions so we could be sure the work was actually being done. With those changes, Haskell began to fall back in speed to more moderate levels. However, in the last few releases of GHC (and the last several rounds of tweaks by the Haskellers on the Cafe list), GHC has risen to the top of the heap, approaching native C in performance overall. What I find quite appealing about this is that the performance is consistent across platforms. While Java's "HotSpot" compiler makes a good showing on Intel, my AMD test machine yields consistently worse results than the Pentium IV system. However, GHC performs equally well on both systems, yielding a similar fractional performance against C in both cases[1,2]. I think this is great because it means that applications written in Haskell can be reasonably expected to yield good performance on all of the common x86 platforms without customizations. Java and several other languages seem to yield varying performance, probably because of advanced optimizations available on the Intel platform that have been hand-coded into those systems. At any rate, I just wanted to write a quick fan letter about Haskell, and let you know how impressed I am with its performance. Looks like it's time for me to start taking a serious look at Haskell for some real-world problems at work now! Best regards, Brent Fulgham http://shootout.alioth.debian.org/ [1] "http://shootout.alioth.debian.org/gp4/benchmark.php? test=all&lang=all" [2] "http://shootout.alioth.debian.org/debian/benchmark.php? test=all&lang=all" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (Darwin) iD8DBQFEnLtkzGDdrzfvUpURAhRBAKCHWiIAJVONyKhDQgG/rBvSmGGDWgCeIufW G2nx/o1ruiXT5Yn9EcNxsFk= =U7WI -----END PGP SIGNATURE-----
Hello Simon, Tuesday, June 27, 2006, 1:44:45 PM, you wrote:
I wanted to write to inform you how shocked I was to see the great advances in performance in the Glorious Haskell Compiler over the last year or so. Of course, we have also benefited from some great contributions by the folks on the Haskell-Cafe mailing list.
it's nice to read but completely false, you know. GHC don't improved it's speed over last year, 6.4.2 differs from 6.4 only in reliability areas source of this great progress is really work of Haskell fans, especially Donald Bruce Stewart. moreover, i quickly scanned the tests where GHC still far behind it's competition and in most cases speed is determined by libraries. when ghc 6.6 with ByteString library included will roll out, we will got higher marks in all those tests except for mandelbrot, which really depends on compiler speed by itself. and in this test we can see Clean and OCaml with gcc-like speed and GHC what is 10 times slower. the only Haskell compiler that will be a great performer here is JHC, i think in all other tests speed is defined by libraries (as i see, Shootout is more a test of bundled libs than compilers) or by hand-optimized code for GHC comparing to the "normal", readable code for other languages. what this proves? first, that libs is exceptionally important for overall speed and interpreted language with fast C-written libs (say, Tcl) may perform better than compiled language like Haskell. second, that the naive implementations are very far from optimum and hand-optimized program in slow language can easily outperform naive program in fast one this great progress in last year says much more about Haskell community and, indirectly, about attractiveness of Haskell. it's a really great language but don't say that GHC and GCC can be compared on their speed ... this remembered me another important question. who is determined which projects to include in Google SoC financing? i read the proposals list and was very pleased by final choice of financed projects - it's really what i consider as most important areas for Haskell infrastructure growth - network, installation, byte strings, data structures. the only two projects not supported that was interesting for me, is about optimization - one about adding something to speed up programs compiled by GHC (i don't remember exactly), second - about further development of JHC. why these projects was not selected? it were no students interested or these projects considered as less important? can i next year participate in discussion/election of projects that will be supported? i want to say especially about JHC. we need to have GHC alternatives - just because competition is best source of evolution. from my POV, jhc is especially interesting as GHC alternative because it's the only compiler oriented to optimized compilation. Hugs, nhc has other directions of evolution (learning and debugging), while adding to jhc modern language extensions should make it real GHC competition (and jhc already generates native C code, so it will have at least one substantial advantage over GHC) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
Bulat Ziganshin wrote:
Tuesday, June 27, 2006, 1:44:45 PM, you wrote:
I wanted to write to inform you how shocked I was to see the great advances in performance in the Glorious Haskell Compiler over the last year or so. Of course, we have also benefited from some great contributions by the folks on the Haskell-Cafe mailing list.
it's nice to read but completely false, you know. GHC don't improved it's speed over last year, 6.4.2 differs from 6.4 only in reliability areas
source of this great progress is really work of Haskell fans, especially Donald Bruce Stewart. moreover, i quickly scanned the tests where GHC still far behind it's competition and in most cases speed is determined by libraries.
This is the great thing. Not only are we already doing quite well, but we know there are substantial improvements still to be had.
.. (and jhc already generates native C code, so it will have at least one substantial advantage over GHC) ...
Compiling via C is a dead end. We realised this about 5 years ago, and yet we still haven't managed to shake off the C backend from GHC, but I remain hopeful that one day we will. /me heads for the cafe... Cheers, Simon
On Tue, Jun 27, 2006 at 02:58:05PM +0100, Simon Marlow wrote:
.. (and jhc already generates native C code, so it will have at least one substantial advantage over GHC) ...
Compiling via C is a dead end. We realised this about 5 years ago, and yet we still haven't managed to shake off the C backend from GHC, but I remain hopeful that one day we will. /me heads for the cafe...
Out of curiosity, compiling via C as opposed to what? c--? Parrot? JVM? I am certainly looking forward to compiling to C-- when it becomes feasable. In any case, jhc's design was not overly influenced by needing to compile to C, compiling to C was easy as a side effect of other design choices. Of course, the advantages of C-- over C arn't as pronounced for jhc as they are for ghc, but there are certainly features I can use. Jhc's performance has been in the dumps recently since I completly broke the strictness analyzer. pretty much no optimizations occur without good strictness transformations. I have been writing a new one using the simple usage polymorphism algorithm, http://research.microsoft.com/~simonpj/Papers/usage-types/usage.htm but with the 2-point strict/notstrict lattice rather than the at most once/many one. something I have noticed is some strong parallels to the other strictness algorithm I played with, the one using HORN constraints. http://citeseer.ist.psu.edu/493889.html in fact, the formulas derived end up pretty much exactly the same if you consider the 2 point lattice to be boolean values and <= to be boolean implication. the act of generalization where you find the variables that occur in both contravarient and covarient positions in the simple usage polymorphism algorithm is similar to the satisfyability problem over boolean formulas. only variables that appear in both positions may be 'pulled strict' by another demanded value. the simple usage analysis algorithm is signfigantly faster, making me think the HORN constraint version is as strong as (and perhaps even equivalent to) constrained polymorphism rather than just simple polymorphism. but this is all speculation.. This also might imply that in order to get good performance extending the simple usage analysis to the full 7-point lattice combining strictenss/abscence/one-shot/linear analysis in one pass, you might need full constrained polymorphism to get good strictness results. Most of this is just idle speculation though based on some perceived parallels... of course, this is probably not very interesting and obvious... but I found it pretty neat. I'm odd like that. Hmm.. what a tangent. John -- John Meacham - ⑆repetae.net⑆john⑈
On Tue, 2006-06-27 at 16:00 +0400, Bulat Ziganshin wrote:
... this remembered me another important question. who is determined which projects to include in Google SoC financing? i read the proposals list and was very pleased by final choice of financed projects - it's really what i consider as most important areas for Haskell infrastructure growth - network, installation, byte strings, data structures. the only two projects not supported that was interesting for me, is about optimization - one about adding something to speed up programs compiled by GHC (i don't remember exactly), second - about further development of JHC. why these projects was not selected? it were no students interested or these projects considered as less important? can i next year participate in discussion/election of projects that will be supported?
The applications were reviewed and ranked by a number of mentors who signed up. Paolo wrote to the haskell list in April to announce that haskell.org was a mentoring organisation and to ask for mentors and project ideas. http://www.haskell.org/pipermail/haskell/2006-April/017898.html http://hackage.haskell.org/trac/summer-of-code/ You are certainly welcome to sign up next year. Duncan
participants (5)
-
Bulat Ziganshin -
Duncan Coutts -
John Meacham -
Simon Marlow -
Simon Marlow