On Fri, Feb 8, 2013 at 1:28 PM, Simon Marlow <marlowsd@gmail.com> wrote:

On 08/02/13 10:10, Nicolas Frisby wrote:

...

Hi Johan. I'm seeing much different numbers when I test for regressions. Also 64-bit Linux. Could you explain your method? Origin of the GHC/libraries you're measuring and the options your running NoFib with? Thanks. I'm using the 28 Jan ghc-7.6.2-x86_64-unknown-**linux.tar.bz2 and ec9377b1e51fd59e18 (Wed Feb 6 14:10:39 2013 +0000) for HEAD. Thanks.

The usual source of differences is -O vs. -O2, either for the benchmarks or for the libraries: you want GhcLibOpts=-O2 and NoFibHcOpts=-O2 in your mk/build.mk file.

Cheers, Simon

I figured Johan is doing O2 for libraries and programs. I have done libraries at O2, programs at O1, which likely does explain the difference. I'm still in a holding pattern in case there's also some other setting he's using. Then I'll run both variations on the programs. Thanks much.

...

On Thu, Feb 7, 2013 at 9:18 PM, Johan Tibell <johan.tibell@gmail.com <mailto:johan.tibell@gmail.com**>> wrote:

Hi all,

I just ran nofib on current HEAD and compared it to 7.6.2 on my 64-bit Linux machine. There are some regressions I think we should look into before a release:

------------------------------**------------------------------** -------------------- Program Size Allocs Runtime Elapsed TotalMem ------------------------------**------------------------------** -------------------- anna -0.8% +1.5% 0.08 0.08 +0.0% ansi -0.5% +0.5% 0.00 0.00 +0.0% atom -0.9% -0.0% -3.1% -0.8% +0.0% awards -0.9% +0.1% 0.00 0.00 +0.0% banner +0.1% +2.2% 0.00 0.00 +0.0% bernouilli -0.6% +2.7% 0.12 0.13 +0.0% boyer -0.7% +0.0% 0.03 0.04 +0.0% boyer2 -0.1% +0.0% 0.01 0.01 +0.0% bspt -0.6% -0.0% 0.01 0.02 +0.0% cacheprof -0.9% +6.9% +0.8% +4.0% +0.0% calendar -0.9% +0.1% 0.00 0.00 +0.0% cichelli -0.1% -0.0% 0.06 0.06 +9.4% circsim -0.7% +0.1% +1.6% +7.0% +13.6% clausify -0.8% +0.0% 0.03 0.03 +0.0% comp_lab_zift -0.7% +0.0% 0.14 0.14 -12.5% compress -0.1% +0.0% 0.12 0.13 +0.0% compress2 -0.0% +0.0% 0.12 0.14 +2.3% constraints -0.8% +0.0% -3.3% -0.7% +0.0% cryptarithm1 -0.1% +0.0% +2.1% +4.1% +0.0% cryptarithm2 -0.0% -0.8% 0.01 0.01 +0.0% cse -0.0% -0.0% 0.00 0.00 +0.0% eliza +0.1% +7.7% 0.00 0.00 +0.0% event -0.8% +0.0% 0.09 0.10 -8.7% exp3_8 -0.8% +0.0% 0.15 0.15 +100.0% expert -0.2% +18.4% 0.00 0.00 +0.0% fem +1.5% +0.5% 0.02 0.02 +0.0% fft -0.9% +0.0% 0.02 0.03 +0.0% fft2 +3.7% +31.9% 0.05 0.05 +20.0% fibheaps -1.0% +0.3% 0.03 0.03 +0.0% fish -0.0% -0.0% 0.01 0.01 +0.0% fluid -1.3% +13.0% 0.01 0.01 +0.0% fulsom -0.2% -0.0% 0.19 0.20 +8.3% gamteb -0.7% -0.2% 0.03 0.04 +0.0% gcd -0.8% +0.0% 0.02 0.03 +0.0% gen_regexps +0.0% +2.2% 0.00 0.00 +0.0% genfft -0.8% -0.3% 0.03 0.03 +0.0% gg -0.6% +40.4% 0.01 0.02 +50.0% grep -1.8% +2.3% 0.00 0.00 +0.0% hidden -0.7% +3.5% +4.1% +7.6% +0.0% hpg -1.3% -1.3% 0.05 0.10 +0.0% ida -0.7% -1.0% 0.07 0.08 +11.1% infer -0.9% +0.5% 0.05 0.05 +30.0% integer -0.8% +1.1% +1.1% +2.3% +0.0% integrate -0.9% +56.2% 0.20 0.23 +1.0% kahan -0.9% +144.9% +77.6% +78.4% +0.0% knights -0.1% -0.4% 0.01 0.01 +0.0% lcss -0.7% +0.7% -24.3% -18.4% +1.9% life -0.2% +0.0% 0.16 0.16 +0.0% lift -0.0% +0.1% 0.00 0.00 +0.0% listcompr -0.1% -0.0% 0.06 0.06 +0.0% listcopy -0.1% -0.0% 0.06 0.06 +0.0% maillist +0.0% +1.9% 0.02 0.04 +16.5% mandel -0.8% +0.0% 0.05 0.05 +0.0% mandel2 -0.1% -4.0% 0.00 0.01 +0.0% minimax -0.2% -0.0% 0.00 0.00 +0.0% mkhprog -0.1% +1.1% 0.00 0.00 +0.0% multiplier -1.2% +0.0% 0.07 0.08 +0.0% nucleic2 -3.3% +19.5% 0.05 0.05 +0.0% para -0.0% +25.0% 0.22 0.23 +0.0% paraffins -0.8% +0.0% 0.06 0.08 +7.5% parser -1.3% +22.2% 0.03 0.03 +0.0% parstof -0.5% +4.6% 0.01 0.00 +0.0% pic +0.5% +0.0% 0.00 0.00 +0.0% power -1.0% +0.0% -0.5% +1.0% +0.0% pretty -0.2% +0.0% 0.00 0.00 +0.0% primes -0.8% -0.0% 0.04 0.05 +0.0% primetest -0.7% +0.0% 0.07 0.07 +0.0% prolog -0.2% +16.0% 0.00 0.00 +0.0% puzzle -0.1% -2.1% 0.09 0.10 +0.0% queens -0.8% +0.0% 0.02 0.02 +0.0% reptile -0.8% +0.6% 0.01 0.02 +0.0% rewrite -0.8% +0.7% 0.02 0.02 +0.0% rfib -1.0% +0.4% 0.02 0.02 +0.0% rsa -0.7% +2.5% 0.02 0.02 +0.0% scc -0.1% +0.0% 0.00 0.00 +0.0% sched -0.8% +0.0% 0.01 0.02 +0.0% scs -1.9% +0.8% -5.3% -2.7% +0.0% simple -0.3% -0.0% 0.15 0.16 +6.9% solid -0.7% +0.0% 0.09 0.09 +0.0% sorting +0.0% +55.9% 0.00 0.00 +0.0% sphere -0.9% -1.8% 0.04 0.04 +0.0% symalg -0.8% +0.3% 0.01 0.01 +0.0% tak -0.8% +0.9% 0.01 0.01 +0.0% transform -0.8% +0.0% -3.3% -5.7% +0.0% treejoin +0.1% +109.3% 0.15 0.17 -7.4% typecheck -0.8% +0.0% 0.14 0.15 +0.0% veritas -0.8% +0.0% 0.00 0.00 +0.0% wang -0.9% +0.0% 0.07 0.08 +0.0% wave4main -0.8% +1.8% 0.18 0.19 -7.1% wheel-sieve1 -0.7% +0.0% +0.0% +1.5% -12.5% wheel-sieve2 -0.7% +0.0% 0.11 0.12 +2.1% x2n1 +7.4% +43.3% 0.01 0.01 +200.0% ------------------------------**------------------------------** -------------------- Min -3.3% -4.0% -24.3% -18.4% -12.5% Max +7.4% +144.9% +77.6% +78.4% +200.0% Geometric Mean -0.5% +5.5% +1.8% +4.3% +3.2%

I haven't had time to look through the regressions yet, so if someone has time, please grab a benchmark that looks bad and have a look at the Core to see what's going on. I suggest starting with "kahan", which is a relatively simple benchmark.

-- Johan

______________________________**_________________ ghc-devs mailing list ghc-devs@haskell.org <mailto:ghc-devs@haskell.org> http://www.haskell.org/**mailman/listinfo/ghc-devs<http://www.haskell.org/mailman/listinfo/ghc-devs>

______________________________**_________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/**mailman/listinfo/ghc-devs<http://www.haskell.org/mailman/listinfo/ghc-devs>

On Fri, Feb 8, 2013 at 8:47 AM, Johan Tibell <johan.tibell@gmail.com> wrote:

If the defaults for nofib are wrong (e.g. if we want -O2 by default) we should tweak the settings so people get sensible results by default.

I've now changed the default to -O2. I should rerun all the benchmarks again with 7.6.1, 7.6.2, and HEAD. -- Johan

I've been running 7.0.1 all through HEAD. I'd like to compare my setup with Johan's before making any claims. On Feb 8, 2013 10:33 AM, "Bryan O'Sullivan" <bos@serpentine.com> wrote:

On Thu, Feb 7, 2013 at 1:18 PM, Johan Tibell <johan.tibell@gmail.com>wrote:

...

I just ran nofib on current HEAD and compared it to 7.6.2 on my 64-bit Linux machine. There are some regressions I think we should look into before a release:

It would be interesting to see the numbers compared to 7.6.1, if possible, as I've seen some big performance regressions between 7.6.1 and 7.6.2. _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

I'm still seeing different numbers from Johan, so I'm just going to throw up my hands and proceed separately for now. I'm sharing my Excel spreadsheet. It contains: * My 8 Feb run (libraries O2, programs O1, 10 iterations) * Johans' first (comparable) set of 7.6.2 -> HEAD numbers (O2, O1, 5 iterations) * My 11 Feb run (libraries O2, programs O2, 30 iterations) * some analysis, indicating that his and my numbers do not correlate Info about my runs: * 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.2.1, 7.2.2, 7.4.1, 7.4.2, 7.6.1, 7.6.2, HEAD (ec9477b1e51) * I using the NoFib make to build the tests, but not to actually execute them. Here's why I don't simply use the NoFib makefiles for everything. That would run an entire suite for each version: all of 7.0.1, then all of 7.0.2, etc. I wanted better time locality, since my test machine (though quite capable - Marlow and SPJ might know the specs), has a few other users and I've been seeing plenty of noise in the measurements. So now I use NoFib's makefiles to build all of the tests and generate the test-running command, but I cache the run command and then interleave the execution of each GHC version's executable. In other words: using nofib: outer loop = GHC version, inner loop = each program my rig: outer loop = each program; inner loop = GHC version NB I do all the building before entering that nested loop. So I think I end up with less noise in the data. I have an additional loop between those two loops that controls the iterations. My rigorous data set (11 Feb) uses 30 iterations. Here are the big swings I'm seeing... Indicated allocation regressions that seem to still affect HEAD: NB The 'swing' column is the subtraction (of percentage points) between HEAD today and the best job GHC ever did on that program. The 'concrete' column converts 'swing' to the actual difference in allocation. 7.0.1 7.0.* 7.2.* 7.4.1 7.4.2 7.6.* HEAD swing concrete atom 1,372,769,928 0.0% 18.5% 18.4% 18.4% 18.4% 18.4% 18% 252,589,667 exp3_8 3,500,231,456 0.0% 68.2% 68.2% 68.2% 68.2% 68.2% 68% 2,387,157,853 integrate 1,092,339,840 0.0% -12.3% -13.8% -57.3% 39.0% -11.0% 46% 505,753,346 lcss 1,272,240,016 0.0% 22.5% 22.5% 22.5% 22.5% 22.2% 22% 282,437,284 mandel 244,936,399 0.0% -9.8% 15.9% 24.1% 24.1% 24.0% 34% 82,788,503 primes 861,817,344 0.0% 57.9% 57.9% 57.9% 57.9% 57.9% 58% 498,992,242 rfib 78,240 4.2% 48.6% 49.7% 48.0% 48.8% 48.2% 48% 37,712 tak 91,304 3.4% 24.7% 24.2% 18.7% 15.8% 14.8% 15% 13,513 wave4main 151,494,392 0.0% 20.3% 22.5% 22.5% 22.5% 20.3% 20% 30,753,362 wheel-sieve1 24,131,096 0.0% -1.6% -1.6% -1.6% 99.2% 99.2% 101% 24,324,145 7.0 to 7.2 and 7.4 to 7.6 seem to be the major players. Indicated allocation regressions that seem to still affect HEAD: 7.0.1 7.0.2 7.0.3 7.0.4 7.2.1 7.2.2 7.4.1 7.4.2 7.6.1 7.6.2 HEAD swing concrete ansi 1.66 0.9% 0.9% -1.0% -0.3% 0.9% -14.1% -16.7% -1.1% 0.5% 5.6% 22.3% 0.37 atom 2.83 1.2% 1.4% -0.2% 8.7% 8.0% 11.3% 9.2% 11.5% 8.7% 9.7% 9.9% 0.28 exp3_8 3.40 0.1% 0.1% -0.9% 19.2% 19.7% 20.6% 23.8% 20.3% 20.4% 19.5% 20.4% 0.69 genfft 0.90 3.1% 3.1% 2.9% 3.1% 2.0% 5.8% 4.5% 3.0% -2.1% 7.4% 9.5% 0.09 hpg 0.28 0.4% 0.1% -34.3% 4.0% -8.6% -11.8% -52.1% -46.0% -51.2% -16.8% 35.3% 0.10 integrate 0.74 -12.2% -12.2% -20.7% -17.0% -14.7% -14.4% -66.7% 58.4% 63.4% -14.5% 52.2% 0.39 knights 1.82 0.2% 0.0% 1.0% 2.8% -3.6% 1.5% 9.0% -7.0% 0.9% 5.1% 12.1% 0.22 lcss 2.81 -0.1% -0.5% -6.0% 6.1% 6.3% 6.4% -0.1% 0.0% -0.4% 24.2% 30.2% 0.85 maillist 0.40 -0.5% 0.7% -63.1% 9.5% 9.7% -10.1% -76.2% -74.0% -75.1% -12.6% 63.6% 0.25 mandel 0.24 -2.6% -2.4% -1.8% -9.7% -11.3% 3.6% 8.3% 10.9% 14.5% 10.4% 21.7% 0.05 wang 1.00 1.4% 1.2% -3.3% 1.0% 2.0% 1.9% -3.7% -1.3% -1.5% 7.0% 10.7% 0.11 More noisy, of course :( The major players from allocation show up again. One new difference is 7.6 -> HEAD, where we seem to lose ~50% in hpg and maillist. I am a bit worried because I see no *huge* swings like Johan reported. Here's an example excerpt from the NoFib makefile's log when building the programs: HC_OPTS = -H32m -O -O -Rghc-timing -package array -H32m -hisuf hi -hisuf hi -hcsuf hc -osuf o -package-db /home/t-nicof/installs/ghc-7.7.20130206/lib/ghc-7.7.20130206/package.conf.d -O2 -rtsopts Does anything look particularly suspicious there? (I used -package-db just to be safe; haven't ran multiple GHC versions too often.) I might investigate some of these myself; it depends on how we allocate my internship time. Please feel free to look into what's happening in some of these! Especially if you have some of the major compiler changes between versions in your head... :) Thanks. HTH. On Fri, Feb 8, 2013 at 8:41 PM, Nicolas Frisby <nicolas.frisby@gmail.com>wrote:

I've been running 7.0.1 all through HEAD. I'd like to compare my setup with Johan's before making any claims. On Feb 8, 2013 10:33 AM, "Bryan O'Sullivan" <bos@serpentine.com> wrote:

On Thu, Feb 7, 2013 at 1:18 PM, Johan Tibell <johan.tibell@gmail.com>wrote:

...

I just ran nofib on current HEAD and compared it to 7.6.2 on my 64-bit Linux machine. There are some regressions I think we should look into before a release:

It would be interesting to see the numbers compared to 7.6.1, if possible, as I've seen some big performance regressions between 7.6.1 and 7.6.2. _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Thanks for such specific requests, and great idea to focus on imaginary! Outline: SUMMARY, HARDWARE, NUMBERS == THE SUMMARY == * I still get quite different numbers. * For wheel-sieve1 and kahan, we're in the same ballpark. * Though my kahan shows half as much allocation increase. Which hardware difference would explain this? * For bernoiulli, exp3_8, and integrate, my nofib-analyse shows percent changes in Runtime, where as yours shows absolutes. I've included my absolute tables below; comparing those, we still get appreciable differences. * I've included my mode=slow numbers. * We have some significant hardware differences. * My machine claims 32 processors, though it smells like it has 8 chips with 8 cores each. I'll ask SPJ or Mainland. * My cache size is much smaller than yours: 512 KB versus 8 MB. * My CPU frequency is 2GHz compared to your 3.4GHz. * How do we want to handle hardware diversity like we're seeing in these regular benchmark runs? * Are the different behaviors we're seeing expected for our hardware differences or bugs of some sort? Thanks, Johan. == THE HARDWARE == $ cat /proc/cpuinfo processor : 0 # counts up to 31, with physical id and core id pairs duplicated once vendor_id : AuthenticAMD cpu family : 16 model : 9 model name : AMD Opteron(tm) Processor 6128 stepping : 1 microcode : 0x10000d4 cpu MHz : 1999.949 cache size : 512 KB physical id : 0 # counts up to 3 for each core id, twice siblings : 8 core id : 0 # counts up to 3, for each physical id, twice cpu cores : 8 apicid : 0 # varies initial apicid : 0 # varies fpu : yes fpu_exception : yes cpuid level : 5 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid amd_dcm pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr npt lbrv svm_lock nrip_save pausefilter bogomips : 3999.89 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: ts ttp tm stc 100mhzsteps hwpstate physical id and core id exhaust combinations of {0,1,2,3}, twice for some reason as processor counts from 0 to 31. I would have suspected 64 processors, given the sibling and cpu cores. Am I tripping on a common misconception? I included the rest of the info because I still get different numbers than you do. $ uname -a Linux cam-05-unx 3.2.0-35-generic #55-Ubuntu SMP Wed Dec 5 17:42:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux $ VERSION=7.0.4; HC=/home/t-nicof/installs/ghc-${VERSION}/bin/ghc-${VERSION}; $HC --info [("Project name","The Glorious Glasgow Haskell Compilation System") ,("Project version","7.0.4") ,("Booter version","6.12.1") ,("Stage","2") ,("Build platform","x86_64-unknown-linux") ,("Host platform","x86_64-unknown-linux") ,("Target platform","x86_64-unknown-linux") ,("Have interpreter","YES") ,("Object splitting","YES") ,("Have native code generator","YES") ,("Have llvm code generator","YES") ,("Support SMP","YES") ,("Unregisterised","NO") ,("Tables next to code","YES") ,("RTS ways","l debug thr thr_debug thr_l thr_p dyn debug_dyn thr_dyn thr_debug_dyn") ,("Leading underscore","NO") ,("Debug on","False") ,("LibDir","/home/t-nicof/installs/ghc-7.0.4/lib/ghc-7.0.4") ,("Global Package DB","/home/t-nicof/installs/ghc-7.0.4/lib/ghc-7.0.4/package.conf.d") ,("C compiler flags","[\"-fno-stack-protector\"]") ,("Gcc Linker flags","[]") ,("Ld Linker flags","[]") $ VERSION=7.6.2; HC=/home/t-nicof/installs/ghc-${VERSION}/bin/ghc-${VERSION}; $HC --info [("Project name","The Glorious Glasgow Haskell Compilation System") ,("GCC extra via C opts"," -fwrapv") ,("C compiler command","/usr/bin/gcc") ,("C compiler flags"," -fno-stack-protector ") ,("ar command","/usr/bin/ar") ,("ar flags","q") ,("ar supports at file","@ArSupportsAtFile@") ,("touch command","touch") ,("dllwrap command","/bin/false") ,("windres command","/bin/false") ,("perl command","/usr/bin/perl") ,("target os","OSLinux") ,("target arch","ArchX86_64") ,("target word size","8") ,("target has GNU nonexec stack","True") ,("target has .ident directive","True") ,("target has subsections via symbols","False") ,("LLVM llc command","llc") ,("LLVM opt command","opt") ,("Project version","7.6.2") ,("Booter version","7.4.1") ,("Stage","2") ,("Build platform","x86_64-unknown-linux") ,("Host platform","x86_64-unknown-linux") ,("Target platform","x86_64-unknown-linux") ,("Have interpreter","YES") ,("Object splitting supported","YES") ,("Have native code generator","YES") ,("Support SMP","YES") ,("Unregisterised","NO") ,("Tables next to code","YES") ,("RTS ways","l debug thr thr_debug thr_l thr_p dyn debug_dyn thr_dyn thr_debug_dyn") ,("Leading underscore","NO") ,("Debug on","False") ,("LibDir","/home/t-nicof/installs/ghc-7.6/lib/ghc-7.6.2") ,("Global Package DB","/home/t-nicof/installs/ghc-7.6/lib/ghc-7.6.2/package.conf.d") ,("Gcc Linker flags","[\"-Wl,--hash-size=31\",\"-Wl,--reduce-memory-overheads\"]") ,("Ld Linker flags","[\"--hash-size=31\",\"--reduce-memory-overheads\"]") ] == THE NUMBERS == With VERSION=7.0.4 or VERSION=7.6.2. (I'm not relying on $PATH, is the only difference.) $ HC=/home/t-nicof/installs/ghc-${VERSION}/bin/ghc-${VERSION}; (make clean && make boot WithNofibHc=${HC} && make WithNofibHc=${HC}) >& log-${VERSION} -------------------------------------------------------------------------------- Program Size Allocs Runtime Elapsed TotalMem -------------------------------------------------------------------------------- bernouilli +3.3% +0.2% +7.2% +7.4% +0.0% exp3_8 +1.1% +53.7% +55.4% +57.5% +300.0% gen_regexps +18.6% +3.9% 0.00 0.00 +0.0% integrate -0.1% +39.0% +110.2% +88.5% +0.0% kahan +1.7% +41.8% +8.2% +8.0% +0.0% paraffins +1.3% -1.2% -3.6% -0.8% +0.0% primes +1.4% +64.7% 0.11 0.11 +50.0% queens +0.8% -0.2% 0.02 0.02 +0.0% rfib +1.7% +42.8% 0.03 0.03 +0.0% tak +0.9% +12.0% 0.02 0.02 +0.0% wheel-sieve1 +1.4% +66.6% -4.0% -4.3% -17.6% wheel-sieve2 +1.4% +0.0% -0.2% -2.1% +0.0% x2n1 +10.3% +41.7% 0.01 0.01 +200.0% -------------------------------------------------------------------------------- Min -0.1% -1.2% -4.0% -4.3% -17.6% Max +18.6% +66.6% +110.2% +88.5% +300.0% Geometric Mean +3.3% +25.6% +19.6% +18.1% +23.0% I did it twice. -------------------------------------------------------------------------------- Program Size Allocs Runtime Elapsed TotalMem -------------------------------------------------------------------------------- bernouilli +3.3% +0.2% +7.0% +7.1% +0.0% exp3_8 +1.1% +53.7% +56.6% +57.8% +300.0% gen_regexps +18.6% +3.9% 0.00 0.00 +0.0% integrate -0.1% +39.0% +102.1% +86.2% +0.0% kahan +1.7% +41.8% +9.5% +8.9% +0.0% paraffins +1.3% -1.2% -0.6% -4.8% +0.0% primes +1.4% +64.7% 0.11 0.11 +50.0% queens +0.8% -0.2% 0.02 0.02 +0.0% rfib +1.7% +42.8% 0.03 0.03 +0.0% tak +0.9% +12.0% 0.02 0.02 +0.0% wheel-sieve1 +1.4% +66.6% -4.4% -4.3% -17.6% wheel-sieve2 +1.4% +0.0% -1.1% -2.8% +0.0% x2n1 +10.3% +41.7% 0.01 0.01 +200.0% -------------------------------------------------------------------------------- Min -0.1% -1.2% -4.4% -4.8% -17.6% Max +18.6% +66.6% +102.1% +86.2% +300.0% Geometric Mean +3.3% +25.6% +19.5% +17.2% +23.0% Maybe your machine is too fast for nofib-analyse to include exp3_8. Allocations ------------------------------------------------------------------------------- Program log-7.0.4 log-7.6.2 ------------------------------------------------------------------------------- bernouilli 303890616 +0.2% exp3_8 389023528 +53.7% gen_regexps 304768 +3.9% integrate 546206856 +39.0% kahan 700842656 +41.8% paraffins 56201680 -1.2% primes 65899520 +64.7% queens 17387888 -0.2% rfib 81176 +42.8% tak 94408 +12.0% wheel-sieve1 14620056 +66.6% wheel-sieve2 88734064 +0.0% x2n1 2491928 +41.7% -1 s.d. ----- +3.0% +1 s.d. ----- +53.2% Average ----- +25.6% Run Time ------------------------------------------------------------------------------- Program log-7.0.4 log-7.6.2 ------------------------------------------------------------------------------- bernouilli 0.28 +7.2% exp3_8 0.21 +55.4% gen_regexps 0.00 0.00 integrate 0.34 +110.2% kahan 1.07 +8.2% paraffins 0.22 -3.6% primes 0.10 0.11 queens 0.02 0.02 rfib 0.03 0.03 tak 0.01 0.02 wheel-sieve1 0.68 -4.0% wheel-sieve2 0.37 -0.2% x2n1 0.00 0.01 -1 s.d. ----- -9.3% +1 s.d. ----- +57.7% Average ----- +19.6% And here are the results using the "mode=slow" Nofib option. Only bernouilli and gen_regexps do not have SLOW_OPTS defined in their Makefile. It's odd that gen_regexps shows such drastic change then... -------------------------------------------------------------------------------- Program Size Allocs Runtime Elapsed TotalMem -------------------------------------------------------------------------------- bernouilli +3.3% +0.2% +6.7% +7.4% +0.0% exp3_8 +1.1% +68.2% +20.2% +20.3% +100.0% gen_regexps +18.6% -1.2% -6.3% -6.0% +0.0% integrate -0.1% +39.0% +114.3% +104.9% +3.9% kahan +1.7% +41.9% +7.5% +7.5% +0.0% paraffins +1.3% -1.2% -3.3% -2.7% +0.3% primes +1.4% +57.9% +0.8% +1.0% +0.0% queens +0.8% -0.2% -1.7% -2.0% +0.0% rfib +1.7% +42.8% +6.3% +6.0% +0.0% tak +0.9% +12.0% -2.4% -2.5% +0.0% wheel-sieve1 +1.4% +99.2% -3.4% -3.4% +58.8% wheel-sieve2 +1.4% -0.1% -3.6% -3.8% +0.0% x2n1 +10.3% +43.1% 0.13 0.13 +1300.0% -------------------------------------------------------------------------------- Min -0.1% -1.2% -6.3% -6.0% +0.0% Max +18.6% +99.2% +114.3% +104.9% +1300.0% Geometric Mean +3.3% +27.4% +8.2% +7.8% +34.3% Allocations ------------------------------------------------------------------------------- Program log-slow-7.0.4 log-slow-7.6.2 ------------------------------------------------------------------------------- bernouilli 303890616 +0.2% exp3_8 3500234960 +68.2% gen_regexps 780759064 -1.2% integrate 1092338624 +39.0% kahan 2797648296 +41.9% paraffins 363166288 -1.2% primes 861820872 +57.9% queens 569243336 -0.2% rfib 81488 +42.8% tak 94408 +12.0% wheel-sieve1 24134568 +99.2% wheel-sieve2 160800936 -0.1% x2n1 19375720 +43.1% -1 s.d. ----- +1.2% +1 s.d. ----- +60.4% Average ----- +27.4% Run Time ------------------------------------------------------------------------------- Program log-slow-7.0.4 log-slow-7.6.2 ------------------------------------------------------------------------------- bernouilli 0.28 +6.7% exp3_8 3.39 +20.2% gen_regexps 1.90 -6.3% integrate 0.68 +114.3% kahan 4.35 +7.5% paraffins 1.93 -3.3% primes 1.51 +0.8% queens 0.72 -1.7% rfib 0.31 +6.3% tak 1.58 -2.4% wheel-sieve1 2.28 -3.4% wheel-sieve2 0.77 -3.6% x2n1 0.04 0.13 -1 s.d. ----- -12.9% +1 s.d. ----- +34.3% Average ----- +8.2% On Tue, Feb 12, 2013 at 3:17 AM, Johan Tibell <johan.tibell@gmail.com>wrote:

Hi Nicolas!

I tried to reproduce the difference between 7.0.4 and 7.6.2 on the exp3_8, wheel-sieve1, and primes and couldn't get the same percent difference as you. We need to reconcile these differences somehow. Lets start with more exact machine specs. I have a:

$ cat /proc/cpuinfo processor : 7 vendor_id : GenuineIntel cpu family : 6 model : 58 model name : Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz stepping : 9 microcode : 0x12 cpu MHz : 1600.000 cache size : 8192 KB ...

$ uname -a Linux johantibell.com 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

And GHC versions:

$ ghc-7.0.4 --info [("Project name","The Glorious Glasgow Haskell Compilation System") ,("Project version","7.0.4") ,("Booter version","6.12.1") ,("Stage","2") ,("Build platform","x86_64-unknown-linux") ,("Host platform","x86_64-unknown-linux") ,("Target platform","x86_64-unknown-linux") ,("Have interpreter","YES") ,("Object splitting","YES") ,("Have native code generator","YES") ,("Have llvm code generator","YES") ,("Support SMP","YES") ,("Unregisterised","NO") ,("Tables next to code","YES") ,("RTS ways","l debug thr thr_debug thr_l thr_p dyn debug_dyn thr_dyn thr_debug_dyn") ,("Leading underscore","NO") ,("Debug on","False") ,("LibDir","/usr/local/lib/ghc-7.0.4") ,("Global Package DB","/usr/local/lib/ghc-7.0.4/package.conf.d") ,("C compiler flags","[\"-fno-stack-protector\"]") ,("Gcc Linker flags","[]") ,("Ld Linker flags","[]") ]

$ ghc-7.6.2 --info [("Project name","The Glorious Glasgow Haskell Compilation System") ,("GCC extra via C opts"," -fwrapv") ,("C compiler command","/usr/bin/gcc") ,("C compiler flags"," -fno-stack-protector ") ,("ar command","/usr/bin/ar") ,("ar flags","q") ,("ar supports at file","@ArSupportsAtFile@") ,("touch command","touch") ,("dllwrap command","/bin/false") ,("windres command","/bin/false") ,("perl command","/usr/bin/perl") ,("target os","OSLinux") ,("target arch","ArchX86_64") ,("target word size","8") ,("target has GNU nonexec stack","True") ,("target has .ident directive","True") ,("target has subsections via symbols","False") ,("LLVM llc command","llc") ,("LLVM opt command","opt") ,("Project version","7.6.2") ,("Booter version","7.4.1") ,("Stage","2") ,("Build platform","x86_64-unknown-linux") ,("Host platform","x86_64-unknown-linux") ,("Target platform","x86_64-unknown-linux") ,("Have interpreter","YES") ,("Object splitting supported","YES") ,("Have native code generator","YES") ,("Support SMP","YES") ,("Unregisterised","NO") ,("Tables next to code","YES") ,("RTS ways","l debug thr thr_debug thr_l thr_p dyn debug_dyn thr_dyn thr_debug_dyn") ,("Leading underscore","NO") ,("Debug on","False") ,("LibDir","/usr/local/lib/ghc-7.6.2") ,("Global Package DB","/usr/local/lib/ghc-7.6.2/package.conf.d") ,("Gcc Linker flags","[\"-Wl,--hash-size=31\",\"-Wl,--reduce-memory-overheads\"]") ,("Ld Linker flags","[\"--hash-size=31\",\"--reduce-memory-overheads\"]") ]

I ran the benchmarks by running e.g.:

$ cd nofib/imaginary/sieve-wheel1 $ make clean && make boot WithNofibHc=ghc-${VERSION} && make WithNofibHc=ghc-${VERSION}

Could you please try to run the "imaginary" benchmarks using exactly these commands and report the difference you see between 7.0.4 and 7.6.2. Here's what I see. 7.0.4 vs 7.6.2:

-------------------------------------------------------------------------------- Program Size Allocs Runtime Elapsed TotalMem

-------------------------------------------------------------------------------- bernouilli +3.3% +0.2% 0.12 0.13 +0.0% exp3_8 +1.1% +53.7% 0.14 0.14 +300.0% gen_regexps +18.7% +3.9% 0.00 0.00 +0.0% integrate -0.1% +39.0% 0.21 0.23 +0.0% kahan +1.7% +98.6% +9.9% +7.3% +0.0% paraffins +1.3% -1.2% 0.06 0.08 +0.0% primes +1.4% +64.7% 0.04 0.05 +50.0% queens +0.8% -0.5% 0.02 0.02 +0.0% rfib +1.7% +42.8% 0.02 0.02 +0.0% tak +0.9% +12.0% 0.01 0.01 +0.0% wheel-sieve1 +0.8% +66.6% -4.6% -5.8% -12.5% wheel-sieve2 +0.9% +0.0% 0.12 0.13 +0.0% x2n1 +10.3% +87.3% 0.00 0.01 +200.0%

-------------------------------------------------------------------------------- Min -0.1% -1.2% -4.6% -5.8% -12.5% Max +18.7% +98.6% +9.9% +7.3% +300.0% Geometric Mean +3.2% +31.7% +2.4% +0.5% +23.6%

-- Johan

I'm using the bindists for some versions, and the ~simonmar/fp/ installs (which I've assumed are bindist) for the ones that already existed there (ie 7.0.1 to 7.4.1, excepting 7.0.4). $ md5sum ghc*bz2 0fc26b4c1d10ff9dc8b0cbe9453d48d3 ghc-7.0.4-x86_64-unknown-linux.tar.bz2 de67ecfe619b0126d8a8b93d26f34555 ghc-7.4.2-x86_64-unknown-linux.tar.bz2 23da3285c5f8fe6716e2795d149c6b96 ghc-7.6.1-x86_64-unknown-linux.tar.bz2 230270a985c522af939d9c71aa76343f ghc-7.6.2-x86_64-unknown-linux.tar.bz2 On Tue, Feb 12, 2013 at 11:09 AM, Simon Marlow <marlowsd@gmail.com> wrote:

On 12/02/13 03:17, Johan Tibell wrote:

...

Hi Nicolas!

I tried to reproduce the difference between 7.0.4 and 7.6.2 on the exp3_8, wheel-sieve1, and primes and couldn't get the same percent difference as you. We need to reconcile these differences somehow. Lets start with more exact machine specs. I have a:

------------------------------**------------------------------** -------------------- Program Size Allocs Runtime Elapsed TotalMem ------------------------------**------------------------------** -------------------- bernouilli +3.3% +0.2% 0.12 0.13 +0.0% exp3_8 +1.1% +53.7% 0.14 0.14 +300.0% gen_regexps +18.7% +3.9% 0.00 0.00 +0.0% integrate -0.1% +39.0% 0.21 0.23 +0.0% kahan +1.7% +98.6% +9.9% +7.3% +0.0% paraffins +1.3% -1.2% 0.06 0.08 +0.0% primes +1.4% +64.7% 0.04 0.05 +50.0% queens +0.8% -0.5% 0.02 0.02 +0.0% rfib +1.7% +42.8% 0.02 0.02 +0.0% tak +0.9% +12.0% 0.01 0.01 +0.0% wheel-sieve1 +0.8% +66.6% -4.6% -5.8% -12.5% wheel-sieve2 +0.9% +0.0% 0.12 0.13 +0.0% x2n1 +10.3% +87.3% 0.00 0.01 +200.0% ------------------------------**------------------------------** -------------------- Min -0.1% -1.2% -4.6% -5.8% -12.5% Max +18.7% +98.6% +9.9% +7.3% +300.0% Geometric Mean +3.2% +31.7% +2.4% +0.5% +23.6%

Some of these benchmarks essentially do no allocation in their inner loops (x2nl, rfib, tak), so differences there just indicate changes in the IO library or elsewhere, and aren't significant.

Is your 7.6.2 from our binary distributions, or did you build it yourself?

Cheers, Simon