On 1/25/07, Neil Mitchell
We have the nofib suite, it could do with some attention, but it is (or was) a pretty good performance tester.
Yes, I've spent enough time staring at nofib results that I understand just how random and un-representative the programs in nofib are. nofib consists of whatever people have submitted for nofib, and moreover, its contents are more or less the same now as they were in 2002. A lot has changed about the ways in which people use Haskell in those past four years -- in 2002, GHC was the only big Haskell application out there, and I don't think that's true anymore. So personally, I have zero faith in nofib results to tell me anything about how helpful optimizations will be except on a very large scale of granularity.
The problem is that something like GHC is very complex, with lots of transformations. When transformations are firing other transformations, which in turn fire other transformations, it doesn't take a great deal to disrupt this flow of optimisation and end up with a result which doesn't accurately reflect the particular change you made. Better knowledge of the end effects on a program isn't necessarily going to translate to better knowledge of the optimisations effect.
Sorry for being unclear. I agree with your comments on GHC, and one thing I was suggesting was that somebody should think about profiling tools for improving our understanding of how those transformations interact with each other, not just profiling tools for understanding the end result. Everyone says that optimization is a black art, but I remain to be convinced that understanding the interactions between different optimizations isn't a problem that would submit to appropriate amounts of effort.
Maybe if we had a greater number and variety of optimising compilers we'd be able to more freely experiment with optimisation techniques in different settings. At the moment GHC is all there is (with Jhc not ready for mainstream use yet)
I agree that there should be more diversity in compilers, but I think even sticking to GHC, there's a lot that could be done when it comes to improving understanding of just what the optimizer is doing. Anything better than staring at intermediate code would be an improvement, since time spent staring at intermediate code usually is time spent narrowing down the 2 lines out of 1000 that are relevant. Maybe it's possible to design tools that could help with that narrowing-down process. Cheers, Kirsten -- Kirsten Chevalier* chevalier@alum.wellesley.edu *Often in error, never in doubt "by God I *KNOW* what this network is for, and you can't have it."--Russ Allbery