Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

On Tue, Feb 5, 2013 at 2:54 AM, Simon Peyton-Jones wrote:

Austin, a 25% performance regression moving to 7.6 is not AT ALL what I expect. I generally expect modest performance improvements? Can you characterise more precisely what is happening? The place I always start is to compile the entire thing with -ticky and see where allocation is changing. (Using -prof affects the optimiser too much.)

I have CC'd Eyal just in case. The discussion was informal but he can hopefully provide more context and rigor. I think off hand, this occurred in a rather large-ish application of his (Lamdu?,) and so tracking down precise reasons may prove difficult. I think the most likely case is just those few 'small cuts' accumulate quickly and are reflecting poorly for this particular case - and that's really the worse 'bug report' of all! Hashable/lens alone for example could certainly make a sizable impact here when added up, e.g. [1] is a recent example of an alleged perf anomaly as of late. And the OS could certainly be relevant.[2] All the more reason to expand nofib and get those bots up! [1] https://github.com/tibbe/hashable/issues/57 [2] Just thinking out loud, but, whenever this happens we really need to characterize results on a per-OS/hardware basis if possible in the future, with some relatively detailed hardware info, to be unambiguous. In terms of raw CPU speed, a lot of benchmarks probably won't stand out due to the OS. But OS X is scheduled to get worse in the SMP case soon[3] for example, and if we inevitably try and start doing things like latency or I/O benchmarks, I'm more than certain things will pop up here. [3] See this ticket: http://hackage.haskell.org/trac/ghc/ticket/7602

Simon

| -----Original Message----- | From: ghc-devs-bounces@haskell.org [mailto:ghc-devs-bounces@haskell.org] | On Behalf Of Austin Seipp | Sent: 05 February 2013 04:22 | To: Johan Tibell | Cc: ghc-devs@haskell.org | Subject: Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2 | | I'm +1 for this. Eyal Lotem and I were just discussing this on IRC a few | minutes ago, and he suffered a rather large (~25%) performance hit when | upgrading to 7.6.1, which is unfortunate. | | Committers are typically very good about recording nofib results in | their commit and being performance-courteous, but I'm not sure there's | ever been a longer-scale view of GHC performance over multiple releases | like this - or even a few months. At least not recently. On top of that, | his application was a type checker, which may certainly stress different | performance points than what nofib might. Once we get performance bots | set up, I've got a small set of machines I'm willing to throw at it. | | Thanks for the results, Johan! | | On Mon, Feb 4, 2013 at 4:33 PM, Johan Tibell | wrote: | > Hi all, | > | > I haven't had much time to do performance tzar work yet, but I did run | > nofib on the last few GHC releases to see the current trend. The | > benchmarks where run on my 64-bit Core i7-3770 @ 3.40GHz Linux | machine. Here are the results: | > | > 7.0.4 to 7.4.2: | > | > ---------------------------------------------------------------------- | ---------- | > Program Size Allocs Runtime Elapsed TotalMem | > ---------------------------------------------------------------------- | ---------- | > Min -1.6% -57.3% -39.1% -36.4% -25.0% | > Max +21.5% +121.5% +24.5% +25.4% +300.0% | > Geometric Mean +8.5% -0.7% -7.1% -5.2% +2.0% | > | > The big loser here in terms of runtime is "kahan", which I added to | > test tight loops involving unboxed arrays and floating point | > arithmetic. I believe there was a regression in fromIntegral RULES | > during this release, which meant that some conversions between | > fixed-width types went via Integer, causing unnecessary allocation. | > | > 7.4.2 to 7.6.1: | > | > ---------------------------------------------------------------------- | ---------- | > Program Size Allocs Runtime Elapsed TotalMem | > ---------------------------------------------------------------------- | ---------- | > Min -5.1% -23.8% -11.8% -12.9% -50.0% | > Max +5.3% +225.5% +7.2% +8.8% +200.0% | > Geometric Mean -0.4% +2.1% +0.3% +0.2% +0.7% | > | > The biggest loser here in terms of runtime is "integrate". I haven't | > looked into why yet. | > | > 7.6.1 to 7.6.2: | > | > ---------------------------------------------------------------------- | ---------- | > Program Size Allocs Runtime Elapsed TotalMem | > ---------------------------------------------------------------------- | ---------- | > Min -2.9% +0.0% -4.8% -4.4% -1.9% | > Max +0.0% +1.0% +4.5% +6.4% +20.8% | > Geometric Mean -1.7% +0.0% +0.1% +0.3% +0.2% | > | > I have two takeaways: | > | > * It's worthwhile running nofib before releases as it does find some | > programs that regressed. | > * There are some other regressions out there (i.e. in code on | > Hackage) that aren't reflected here, suggesting that we need to add | > more programs to nofib. | > | > Cheers, | > Johan | > | > | > _______________________________________________ | > ghc-devs mailing list | > ghc-devs@haskell.org | > http://www.haskell.org/mailman/listinfo/ghc-devs | > | | | | -- | Regards, | Austin | | _______________________________________________ | ghc-devs mailing list | ghc-devs@haskell.org | http://www.haskell.org/mailman/listinfo/ghc-devs

-- Regards, Austin