RE: can a lazy language give fast code?
Can one write withthe Haskell compliler faster code than in the examples of http://www.bagley.org/~doug/shootout/ where GHC (old Haskell 98?) seems to be much slower than Ocaml or Mlton both strict functional languages. Can one expect any improvements in speed in the future?
Many of those programs can be written differently to improve performance. One issue that is affecting performance in several cases is the speed of character I/O, and the representation of Strings as lists of characters. Dramatic improvements can be had by using unboxed arrays of Char (Data.Array.Unboxed), or PackedString (which these days is implemented using unboxed arrays in GHC). Cheers, Simon
What I also meant but did not write was this: is there anyone who would like
to redo these benchmarks and see what it gives with all the new inventions
the DHC supports?
Cheers
Scott
----- Original Message -----
From: "Simon Marlow"
Can one write withthe Haskell compliler faster code than in the examples of http://www.bagley.org/~doug/shootout/ where GHC (old Haskell 98?) seems to be much slower than Ocaml or Mlton both strict functional languages. Can one expect any improvements in speed in the future?
Many of those programs can be written differently to improve performance. One issue that is affecting performance in several cases is the speed of character I/O, and the representation of Strings as lists of characters. Dramatic improvements can be had by using unboxed arrays of Char (Data.Array.Unboxed), or PackedString (which these days is implemented using unboxed arrays in GHC).
Cheers, Simon _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
G'day all. On Mon, Jul 29, 2002 at 10:23:05AM +0100, Simon Marlow wrote:
Many of those programs can be written differently to improve performance.
To be fair, Doug admits this as well as a lot more: http://www.bagley.org/~doug/shootout/method.shtml#flaws Despite these flaws, I did notice that ghc is right in the middle in his CRAPS score system (which is really interesting; all due respect to the GHC guys, but I expected it to be lower <g>). I also noticed that the majority of cases where Haskell does significantly worse than average are "same way" tests, designed to test the performance of various constructs (e.g. array access, dictionary lookup) as opposed to "same thing" tests, designed to test native idioms. In the end, though, benchmarks ignore one of the most important rules of software performance: "throughput" (i.e. the amount of processing that your system can do just prior to being overloaded) is almost never the most important consideration. Other considerations such as flexibility, robustness, responsiveness and scalability are almost always more important. I've thought for a while that what we need is more benchmarks like pseudoknot: Real tasks which real people want to do. Computing Ackermann's function is all well and good, but when's the last time you actually needed to compute it in a real program? Off the top of my head, some "real" tasks which could be benchmarked include: - MPEG video compression. - Code scheduling/register allocation for a CPU like the MIPS/Alpha or even the IA64. - Fluid simulation. - Solving chess problems. Cheers, Andrew Bromage
On Tue, 30 Jul 2002, Andrew J Bromage wrote: [snip]
In the end, though, benchmarks ignore one of the most important rules of software performance: "throughput" (i.e. the amount of processing that your system can do just prior to being overloaded) is almost never the most important consideration. Other considerations such as flexibility, robustness, responsiveness and scalability are almost always more important.
Mmm, such statements really assume that there's a sensible meaning to `almost always' when applied to the set of all programmers, whereas I think a much more realistic assumption is that `there's lots of people out there, all with different priorities' and present things in way which lets people perform their own evaluations. (In cases where I've reason to believe how I code I can simply, reliably and significantly affect throughput I care very much about it.) The problem with language benchmarks is not that they `over-rate' the importance of performance but that they assume per se that choice of language is a single-variable (execution speed) optimization problem; there's no attempt to measure the other items in your list, most especially flexibility. (I'm assuming you mean flexibility of the programmer rewriting, retargeting, refactoring and re-engineering exisiting code.) Of course, I don't have any good ideas about how to measure these, particularly flexibility, in a practicallty implementable and accurate way :-)
I've thought for a while that what we need is more benchmarks like pseudoknot: Real tasks which real people want to do. Computing Ackermann's function is all well and good, but when's the last time you actually needed to compute it in a real program?
I suspect there probably are things that make Ackermann's function a bad `test-case' (eg, computationally simple and regular => good cache utilisation after optimzation which don't extrapolate?) but, for the purposes I'd want to use benchmarks for these deficiencies compared to things like BLAS performance -- which are things that _some_ real people do all day -- are probably don't affect the results all that much. Of more concern to me is, when's the last time you actually got a well specified computational problem and a reasonable amount of time to write a carefully crafted program to solve it, (particularly when you had some reassurance that the very specification of what to solve wouldn't change after the first time you ran the code :-) )?
Off the top of my head, some "real" tasks which could be benchmarked include:
- MPEG video compression.
Here you really want to measure `glue-code' overhead (both performance wise and `human rewriting'-wise) of linking together core processing elements either written in low-level code (MMX, etc) or available on DSP chips, in a way which would allow shifting components and algorithms about. (I.e., in my opinion it's an informative task to have `benchmark-type' data about because it's complicated with many ways to solve the problem, not because it's `real world'.) ___cheers,_dave_________________________________________________________ www.cs.bris.ac.uk/~tweed/ | `It's no good going home to practise email:tweed@cs.bris.ac.uk | a Special Outdoor Song which Has To Be work tel:(0117) 954-5250 | Sung In The Snow' -- Winnie the Pooh
G'day all. On Tue, Jul 30, 2002 at 08:14:27AM +0100, D. Tweed wrote:
Mmm, such statements really assume that there's a sensible meaning to `almost always' when applied to the set of all programmers, whereas I think a much more realistic assumption is that `there's lots of people out there, all with different priorities' and present things in way which lets people perform their own evaluations.
Let me clarify what I meant by that and see if you still disagree. Realistically, _most_ new software installations today (I deliberately ignore legacy systems etc) are not overloaded, in that there are more "computrons" available than are required to perform the task required of them. Of course this is partly because when you do a new installation, you add more than you need because you expect to grow. Secondly, most non-embedded CPUs in the world are not overloaded either. Chances are for a given desktop machine, it spends most of its waking hours waiting for the next keystroke or mouse movement. Web developers in particular know this: For the typical case, your application server runs at the speed of the network. If you agree with me so far, it follows that for most _new_ software, "throughput" is not as great a consideration as other performance metrics, because "throughput" measures the saturation point of your system, and most new systems don't get saturated. Of course I'm speaking in generalities, and there are an awful lot of situations where throughput is an issue. I've worked in a few of those situations before. I'm working in that situation right now, in fact. Throughput measures are important to have if this is the situation that you're in. More information is good. Perhaps the problem is I don't trust everyone to use the information wisely?
The problem with language benchmarks is not that they `over-rate' the importance of performance but that they assume per se that choice of language is a single-variable (execution speed) optimization problem; there's no attempt to measure the other items in your list, most especially flexibility.
While I agree with you here, I don't even claim that language benchmarks of this kind "over-rate" the value of performance. I claim that they don't measure "performance" _at_all_! They measure (in this case) two possible measures of performance, namely memory usage and execution speed, but ignores factors like scalability and latency, which are IMO often more important.
Of more concern to me is, when's the last time you actually got a well specified computational problem and a reasonable amount of time to write a carefully crafted program to solve it, (particularly when you had some reassurance that the very specification of what to solve wouldn't change after the first time you ran the code :-) )?
Perhaps the ICFP contests are actually fairer as benchmarks than as competitions? Cheers, Andrew Bromage
On Wed, 31 Jul 2002, Andrew J Bromage wrote:
Let me clarify what I meant by that and see if you still disagree.
Realistically, _most_ new software installations today (I deliberately ignore legacy systems etc) are not overloaded, in that there are more "computrons" available than are required to perform the task required of them. Of course this is partly because when you do a new installation, you add more than you need because you expect to grow.
I don't disagree at all with your conclusion that there are many factors other than throughput that a programmer wants to know and trade-off when choosing a language. It's in saying this is warranted by `almost all' processes being bound by things other than throughput which may be true in the average sense, but I don't think that all programmers have almost all their programming tasks being dominated by something other than raw throughput but rather there are sets of programmers who have all of the tasks being dominated by the need something else (robustness, say) and some who have all their tasks being dominated by the need for raw throughput. To an extent, I'm being pedantic but I do think it's important when re-thinking benchmarks to recognise that it's a diverse world of programming out there and ideally we want programmers to be able to perform comparisons between languages using the criteria that matter to them (and some may validly value throughput) rather than to change from measuring on only one variable (throughput) to measuring on a different variable but still only one variable.
Secondly, most non-embedded CPUs in the world are not overloaded either. Chances are for a given desktop machine, it spends most of its waking hours waiting for the next keystroke or mouse movement. Web developers in particular know this: For the typical case, your application server runs at the speed of the network.
This is a perfect example of where using an average is pretty misleading: my desktop machine spends a maybe half of its time doing essentially nothing since my thinking time as I write programs and papers is long enough that the text editor, etc, spends most of its time waiting on input. The other half the time it's running image processing code which is essentially CPU bound, so it's running at close to 100% processor utilisation. But (even one of the robust-statistics definitions of) the average would say my machine is using about half the processor power at any given time instant. Clearly this isn't what's happening, there's actually two regimes of operation which it switches between. You make very good points in what I've snipped below, again it's just the use of `most' in a way that implies (to me) taking an average as the representative of what everyone has to deal with that I `disagree with'. [snip]
Of more concern to me is, when's the last time you actually got a well specified computational problem and a reasonable amount of time to write a carefully crafted program to solve it, (particularly when you had some reassurance that the very specification of what to solve wouldn't change after the first time you ran the code :-) )?
Perhaps the ICFP contests are actually fairer as benchmarks than as competitions?
Interesting thought, particularly if the judges announced changes to what the problem to be solved was half-way through :-) ___cheers,_dave_________________________________________________________ www.cs.bris.ac.uk/~tweed/ | `It's no good going home to practise email:tweed@cs.bris.ac.uk | a Special Outdoor Song which Has To Be work tel:(0117) 954-5250 | Sung In The Snow' -- Winnie the Pooh
Hi,
I don't think I have got a fair answer to my questions regarding these
(silly?) benchmarks. I cannot write the programs with the unboxed "things",
but I have both the Ocaml compiler and the latest Glasgow compiler installed
on my windows XP machine. So, if someone sends the programs I'll type it in
and let you know these results. I don't want to be impolite : the fact that
I am on this list proves that I am seriously interested in the elegance of
Haskell. But I am searching a language to program in it: I think e.g. to a
front end of the Lout typesetting program. Also I have the impression that
such fancy things as HOpenGL are not for windows because of the GTK
bindings. It seems that I have to move to a Linux OS.
Regards
John Scott
----- Original Message -----
From: "D. Tweed"
On Wed, 31 Jul 2002, Andrew J Bromage wrote:
Let me clarify what I meant by that and see if you still disagree.
Realistically, _most_ new software installations today (I deliberately ignore legacy systems etc) are not overloaded, in that there are more "computrons" available than are required to perform the task required of them. Of course this is partly because when you do a new installation, you add more than you need because you expect to grow.
I don't disagree at all with your conclusion that there are many factors other than throughput that a programmer wants to know and trade-off when choosing a language. It's in saying this is warranted by `almost all' processes being bound by things other than throughput which may be true in the average sense, but I don't think that all programmers have almost all their programming tasks being dominated by something other than raw throughput but rather there are sets of programmers who have all of the tasks being dominated by the need something else (robustness, say) and some who have all their tasks being dominated by the need for raw throughput. To an extent, I'm being pedantic but I do think it's important when re-thinking benchmarks to recognise that it's a diverse world of programming out there and ideally we want programmers to be able to perform comparisons between languages using the criteria that matter to them (and some may validly value throughput) rather than to change from measuring on only one variable (throughput) to measuring on a different variable but still only one variable.
Secondly, most non-embedded CPUs in the world are not overloaded either. Chances are for a given desktop machine, it spends most of its waking hours waiting for the next keystroke or mouse movement. Web developers in particular know this: For the typical case, your application server runs at the speed of the network.
This is a perfect example of where using an average is pretty misleading: my desktop machine spends a maybe half of its time doing essentially nothing since my thinking time as I write programs and papers is long enough that the text editor, etc, spends most of its time waiting on input. The other half the time it's running image processing code which is essentially CPU bound, so it's running at close to 100% processor utilisation. But (even one of the robust-statistics definitions of) the average would say my machine is using about half the processor power at any given time instant. Clearly this isn't what's happening, there's actually two regimes of operation which it switches between.
You make very good points in what I've snipped below, again it's just the use of `most' in a way that implies (to me) taking an average as the representative of what everyone has to deal with that I `disagree with'.
[snip]
Of more concern to me is, when's the last time you actually got a well specified computational problem and a reasonable amount of time to write a carefully crafted program to solve it, (particularly when you had some reassurance that the very specification of what to solve wouldn't change after the first time you ran the code :-) )?
Perhaps the ICFP contests are actually fairer as benchmarks than as competitions?
Interesting thought, particularly if the judges announced changes to what the problem to be solved was half-way through :-)
___cheers,_dave_________________________________________________________ www.cs.bris.ac.uk/~tweed/ | `It's no good going home to practise email:tweed@cs.bris.ac.uk | a Special Outdoor Song which Has To Be work tel:(0117) 954-5250 | Sung In The Snow' -- Winnie the Pooh
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
On Wed, 31 Jul 2002, Scott J. wrote:
I don't think I have got a fair answer to my questions regarding these (silly?) benchmarks. I cannot write the programs with the unboxed "things", but I have both the Ocaml compiler and the latest Glasgow compiler installed on my windows XP machine. So, if someone sends the programs I'll type it in and let you know these results. I don't want to be impolite : the fact that I am on this list proves that I am seriously interested in the elegance of Haskell. But I am searching a language to program in it: I think e.g. to a front end of the Lout typesetting program. Also I have the impression that such fancy things as HOpenGL are not for windows because of the GTK bindings. It seems that I have to move to a Linux OS.
My messages were more addressing the point which came up about what the aims of benchmarking `ought to be' rather than addressing the question. It seems to me most of the most responses are to the question `could a lazy language compiler be written to give fast code' and you're looking at the question `are there settings and programming idioms for a current compiler that give fast code'. I'll leave it up to the much better qualified various experts with the various compilers to give detailled advice. ___cheers,_dave_________________________________________________________ www.cs.bris.ac.uk/~tweed/ | `It's no good going home to practise email:tweed@cs.bris.ac.uk | a Special Outdoor Song which Has To Be work tel:(0117) 954-5250 | Sung In The Snow' -- Winnie the Pooh
"Scott J."
I don't think I have got a fair answer to my questions regarding these (silly?) benchmarks. I cannot write the programs with the unboxed "things", but I have both the Ocaml compiler and the latest Glasgow compiler installed on my windows XP machine. So, if someone sends the programs I'll type it in and let you know these results. I don't want to be impolite : the fact that I am on this list proves that I am seriously interested in the elegance of Haskell. But I am searching a language to program in it: I think e.g. to a front end of the Lout typesetting program. Also I have the impression that such fancy things as HOpenGL are not for windows because of the GTK bindings.
HOpenGL runs under windows. The GTK+ binding will too after it has been moved over to track GTK+ 2.0.
It seems that I have to move to a Linux OS.
I think that this is generally a very advisable thing to do :-) Cheers, Manuel
On Wed, 31 Jul 2002, Andrew J Bromage wrote:
Perhaps the ICFP contests are actually fairer as benchmarks than as competitions?
On Wed, Jul 31, 2002 at 09:59:31AM +0100, D. Tweed wrote:
Interesting thought, particularly if the judges announced changes to what the problem to be solved was half-way through :-)
Don't forget to change the operating system, programming language, hardware platform, work site, and lay off 80% of the team. =) Cheers, Bill
G'day all. On Wed, Jul 31, 2002 at 09:59:31AM +0100, D. Tweed wrote:
It's in saying this is warranted by `almost all' processes being bound by things other than throughput which may be true in the average sense, but I don't think that all programmers have almost all their programming tasks being dominated by something other than raw throughput but rather there are sets of programmers who have all of the tasks being dominated by the need something else (robustness, say) and some who have all their tasks being dominated by the need for raw throughput.
Fair enough. I was speaking in generalities and average cases and deliberately avoiding the special needs of many programmers and applications. I've worked in enterprise applications, web applications and high-performance servers (both CPU-bound and I/O-bound) and the concerns of all of them are different. Perhaps if I cut down on the superlatives I can say something that everyone agrees with: An awful lot of new code today is written in languages like Visual Basic and Java, despite their relative unsuitability for high "throughput". If it doesn't stop the use of those kinds of languages, it shouldn't stop the use of Haskell either, especially since Haskell arguably scales much better. Therefore were I someone with a stake in the future of Haskell, I would not be not overly concerned about these kinds of benchmarks. Speed is important, and it should be worked on, but it's not as important as the things which Haskell already does better.
You make very good points in what I've snipped below, again it's just the use of `most' in a way that implies (to me) taking an average as the representative of what everyone has to deal with that I `disagree with'.
Sure. I wasn't implying that it was representative of what everyone has to deal with. It's certainly not representative of what I do for a living, though it's pretty close to something I used to do. Perhaps the quibble is over the word "average". While I don't think I used that word, if I did, I'd mean "mode" rather than "mean". :-) Cheers, Andrew Bromage
participants (6)
-
Andrew J Bromage -
D. Tweed -
Manuel M T Chakravarty -
Scott J. -
Simon Marlow -
William Lee Irwin III