[GHC] #15357: Make nofib suitable for runtime measurements.
#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: (none) Type: task | Status: new Priority: normal | Milestone: 8.6.1 Component: NoFib | Version: 8.4.3 benchmark suite | Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Currently many benchmarks in a default nofib run have runtimes so low that they become meaningless. This comes with a bunch of issues: * It takes up build/runtime while not actually improving the accuracy of runtime measurements. * It skews results as the jump from 1ms to 2ms registers as a 200% change. * It can hide regression which fall outside of the measured granularity. * Higher number of runs per benchmark spend far too much time on the few slower benchmarks. At the extreme end there are 9 benchmarks with runtimes shorter than the granularity of 1ms. For some benchmarks changing this might be as easy as increasing the number of runs/iterations. For others changing problem sizes is not as easy since they process more complex input data. Below are the runtimes of a recent default nofib run. ||= benchmark =|| Time (ms) || || ansi || 0 || || awards || 0 || || banner || 0 || || calendar || 0 || || expert || 0 || || gen_regexps || 0 || || grep || 0 || || pretty || 0 || || scc || 0 || || cse || 1 || || eliza || 1 || || lift || 1 || || mkhprog || 1 || || prolog || 1 || || sorting || 1 || || veritas || 1 || || knights || 2 || || minimax || 2 || || x2n1 || 2 || || mandel2 || 3 || || parstof || 3 || || fluid || 4 || || pic || 4 || || boyer2 || 5 || || bspt || 5 || || cryptarithm2 || 5 || || fish || 6 || || gg || 6 || || reptile || 6 || || symalg || 6 || || tak || 6 || || VSD || 7 || || rfib || 7 || || queens || 9 || || rewrite || 9 || || sched || 11 || || fem || 12 || || fibheaps || 13 || || parser || 14 || || rsa || 14 || || genfft || 15 || || clausify || ansi || 0 || || awards || 0 || || banner || 0 || || calendar || 0 || || expert || 0 || || gen_regexps || 0 || || grep || 0 || || pretty || 0 || || scc || 0 || || cse || 1 || || eliza || 1 || || lift || 1 || || mkhprog || 1 || || prolog || 1 || || sorting || 1 || || veritas || 1 || || knights || 2 || || minimax || 2 || || x2n1 || 2 || || mandel2 || 3 || || parstof || 3 || || fluid || 4 || || pic || 4 || || boyer2 || 5 || || bspt || 5 || || cryptarithm2 || 5 || || fish || 6 || || gg || 6 || || reptile || 6 || || symalg || 6 || || tak || 6 || || VSD || 7 || || rfib || 7 || || queens || 9 || || rewrite || 9 || || sched || 11 || || fem || 12 || || fibheaps || 13 || || parser || 14 || || rsa || 14 || || genfft || 15 || || clausify || 17 || || gamteb || 17 || || fft || 18 || || boyer || 20 || || gcd || 20 || || sphere || 21 || || fft2 || 23 || || infer || 25 || || maillist || 25 || || mandel || 27 || || nucleic2 || 30 || || primes || 34 || || cichelli || 35 || || ida || 39 || || listcompr || 41 || || listcopy || 43 || || multiplier || 43 || || wang || 43 || || hpg || 45 || || anna || 46 || || reverse-complem || 46 || || integrate || 48 || || primetest || 50 || || paraffins || 54 || || solid || 54 || || puzzle || 55 || || treejoin || 61 || || compress2 || 64 || || compress || 65 || || event || 65 || || bernouilli || 74 || || comp_lab_zift || 78 || || simple || 86 || || wheel-sieve2 || 91 || || life || 100 || || exp3_8 || 102 || || wave4main || 103 || || typecheck || 106 || || atom || 110 || || fulsom || 112 || || CS || 116 || || para || 120 || || kahan || 138 || || transform || 145 || || power || 150 || || cacheprof || 151 || || hidden || 160 || || scs || 168 || || lcss || 175 || || wheel-sieve1 || 199 || || VS || 204 || || pidigits || 216 || || FS || 222 || || last-piece || 230 || || fasta || 263 || || cryptarithm1 || 264 || || CSD || 272 || || digits-of-e1 || 273 || || binary-trees || 342 || || digits-of-e2 || 381 || || circsim || 418 || || S || 451 || || VSM || 497 || || lambda || 617 || || integer || 905 || || n-body || 1043 || || linear || 1070 || || spectral-norm || 1132 || || constraints || 1171 || || exact-reals || 1289 || || mate || 1955 || || fannkuch-redux || 2185 || || k-nucleotide || 2989 || || 17 || || gamteb || 17 || || fft || 18 || || boyer || 20 || || gcd || 20 || || sphere || 21 || || fft2 || 23 || || infer || 25 || || maillist || 25 || || mandel || 27 || || nucleic2 || 30 || || primes || 34 || || cichelli || 35 || || ida || 39 || || listcompr || 41 || || listcopy || 43 || || multiplier || 43 || || wang || 43 || || hpg || 45 || || anna || 46 || || reverse-complem || 46 || || integrate || 48 || || primetest || 50 || || paraffins || 54 || || solid || 54 || || puzzle || 55 || || treejoin || 61 || || compress2 || 64 || || compress || 65 || || event || 65 || || bernouilli || 74 || || comp_lab_zift || 78 || || simple || 86 || || wheel-sieve2 || 91 || || life || 100 || || exp3_8 || 102 || || wave4main || 103 || || typecheck || 106 || || atom || 110 || || fulsom || 112 || || CS || 116 || || para || 120 || || kahan || 138 || || transform || 145 || || power || 150 || || cacheprof || 151 || || hidden || 160 || || scs || 168 || || lcss || 175 || || wheel-sieve1 || 199 || || VS || 204 || || pidigits || 216 || || FS || 222 || || last-piece || 230 || || fasta || 263 || || cryptarithm1 || 264 || || CSD || 272 || || digits-of-e1 || 273 || || binary-trees || 342 || || digits-of-e2 || 381 || || circsim || 418 || || S || 451 || || VSM || 497 || || lambda || 617 || || integer || 905 || || n-body || 1043 || || linear || 1070 || || spectral-norm || 1132 || || constraints || 1171 || || exact-reals || 1289 || || mate || 1955 || || fannkuch-redux || 2185 || || k-nucleotide || 2989 || -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: (none) Type: task | Status: new Priority: normal | Milestone: 8.6.1 Component: NoFib benchmark | Version: 8.4.3 suite | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Old description:
Currently many benchmarks in a default nofib run have runtimes so low that they become meaningless.
This comes with a bunch of issues: * It takes up build/runtime while not actually improving the accuracy of runtime measurements. * It skews results as the jump from 1ms to 2ms registers as a 200% change. * It can hide regression which fall outside of the measured granularity. * Higher number of runs per benchmark spend far too much time on the few slower benchmarks.
At the extreme end there are 9 benchmarks with runtimes shorter than the granularity of 1ms.
For some benchmarks changing this might be as easy as increasing the number of runs/iterations. For others changing problem sizes is not as easy since they process more complex input data.
Below are the runtimes of a recent default nofib run.
||= benchmark =|| Time (ms) || || ansi || 0 || || awards || 0 || || banner || 0 || || calendar || 0 || || expert || 0 || || gen_regexps || 0 || || grep || 0 || || pretty || 0 || || scc || 0 || || cse || 1 || || eliza || 1 || || lift || 1 || || mkhprog || 1 || || prolog || 1 || || sorting || 1 || || veritas || 1 || || knights || 2 || || minimax || 2 || || x2n1 || 2 || || mandel2 || 3 || || parstof || 3 || || fluid || 4 || || pic || 4 || || boyer2 || 5 || || bspt || 5 || || cryptarithm2 || 5 || || fish || 6 || || gg || 6 || || reptile || 6 || || symalg || 6 || || tak || 6 || || VSD || 7 || || rfib || 7 || || queens || 9 || || rewrite || 9 || || sched || 11 || || fem || 12 || || fibheaps || 13 || || parser || 14 || || rsa || 14 || || genfft || 15 || || clausify || ansi || 0 || || awards || 0 || || banner || 0 || || calendar || 0 || || expert || 0 || || gen_regexps || 0 || || grep || 0 || || pretty || 0 || || scc || 0 || || cse || 1 || || eliza || 1 || || lift || 1 || || mkhprog || 1 || || prolog || 1 || || sorting || 1 || || veritas || 1 || || knights || 2 || || minimax || 2 || || x2n1 || 2 || || mandel2 || 3 || || parstof || 3 || || fluid || 4 || || pic || 4 || || boyer2 || 5 || || bspt || 5 || || cryptarithm2 || 5 || || fish || 6 || || gg || 6 || || reptile || 6 || || symalg || 6 || || tak || 6 || || VSD || 7 || || rfib || 7 || || queens || 9 || || rewrite || 9 || || sched || 11 || || fem || 12 || || fibheaps || 13 || || parser || 14 || || rsa || 14 || || genfft || 15 || || clausify || 17 || || gamteb || 17 || || fft || 18 || || boyer || 20 || || gcd || 20 || || sphere || 21 || || fft2 || 23 || || infer || 25 || || maillist || 25 || || mandel || 27 || || nucleic2 || 30 || || primes || 34 || || cichelli || 35 || || ida || 39 || || listcompr || 41 || || listcopy || 43 || || multiplier || 43 || || wang || 43 || || hpg || 45 || || anna || 46 || || reverse-complem || 46 || || integrate || 48 || || primetest || 50 || || paraffins || 54 || || solid || 54 || || puzzle || 55 || || treejoin || 61 || || compress2 || 64 || || compress || 65 || || event || 65 || || bernouilli || 74 || || comp_lab_zift || 78 || || simple || 86 || || wheel-sieve2 || 91 || || life || 100 || || exp3_8 || 102 || || wave4main || 103 || || typecheck || 106 || || atom || 110 || || fulsom || 112 || || CS || 116 || || para || 120 || || kahan || 138 || || transform || 145 || || power || 150 || || cacheprof || 151 || || hidden || 160 || || scs || 168 || || lcss || 175 || || wheel-sieve1 || 199 || || VS || 204 || || pidigits || 216 || || FS || 222 || || last-piece || 230 || || fasta || 263 || || cryptarithm1 || 264 || || CSD || 272 || || digits-of-e1 || 273 || || binary-trees || 342 || || digits-of-e2 || 381 || || circsim || 418 || || S || 451 || || VSM || 497 || || lambda || 617 || || integer || 905 || || n-body || 1043 || || linear || 1070 || || spectral-norm || 1132 || || constraints || 1171 || || exact-reals || 1289 || || mate || 1955 || || fannkuch-redux || 2185 || || k-nucleotide || 2989 || || 17 || || gamteb || 17 || || fft || 18 || || boyer || 20 || || gcd || 20 || || sphere || 21 || || fft2 || 23 || || infer || 25 || || maillist || 25 || || mandel || 27 || || nucleic2 || 30 || || primes || 34 || || cichelli || 35 || || ida || 39 || || listcompr || 41 || || listcopy || 43 || || multiplier || 43 || || wang || 43 || || hpg || 45 || || anna || 46 || || reverse-complem || 46 || || integrate || 48 || || primetest || 50 || || paraffins || 54 || || solid || 54 || || puzzle || 55 || || treejoin || 61 || || compress2 || 64 || || compress || 65 || || event || 65 || || bernouilli || 74 || || comp_lab_zift || 78 || || simple || 86 || || wheel-sieve2 || 91 || || life || 100 || || exp3_8 || 102 || || wave4main || 103 || || typecheck || 106 || || atom || 110 || || fulsom || 112 || || CS || 116 || || para || 120 || || kahan || 138 || || transform || 145 || || power || 150 || || cacheprof || 151 || || hidden || 160 || || scs || 168 || || lcss || 175 || || wheel-sieve1 || 199 || || VS || 204 || || pidigits || 216 || || FS || 222 || || last-piece || 230 || || fasta || 263 || || cryptarithm1 || 264 || || CSD || 272 || || digits-of-e1 || 273 || || binary-trees || 342 || || digits-of-e2 || 381 || || circsim || 418 || || S || 451 || || VSM || 497 || || lambda || 617 || || integer || 905 || || n-body || 1043 || || linear || 1070 || || spectral-norm || 1132 || || constraints || 1171 || || exact-reals || 1289 || || mate || 1955 || || fannkuch-redux || 2185 || || k-nucleotide || 2989 ||
New description: Currently many benchmarks in a default nofib run have runtimes so low that they become meaningless. This comes with a bunch of issues: * It takes up build/runtime while not actually improving the accuracy of runtime measurements. * It skews results as the jump from 1ms to 2ms registers as a 200% change. * It can hide regression which fall outside of the measured granularity. * Higher number of runs per benchmark spend far too much time on the few slower benchmarks. At the extreme end there are 9 benchmarks with runtimes shorter than the granularity of 1ms. For some benchmarks changing this might be as easy as increasing the number of runs/iterations. For others changing problem sizes is not as easy since they process more complex input data. Below are the runtimes of a recent default nofib run. ||= benchmark =|| Time (ms) || || ansi || 0 || || awards || 0 || || banner || 0 || || calendar || 0 || || expert || 0 || || gen_regexps || 0 || || grep || 0 || || pretty || 0 || || scc || 0 || || cse || 1 || || eliza || 1 || || lift || 1 || || mkhprog || 1 || || prolog || 1 || || sorting || 1 || || veritas || 1 || || knights || 2 || || minimax || 2 || || x2n1 || 2 || || mandel2 || 3 || || parstof || 3 || || fluid || 4 || || pic || 4 || || boyer2 || 5 || || bspt || 5 || || cryptarithm2 || 5 || || fish || 6 || || gg || 6 || || reptile || 6 || || symalg || 6 || || tak || 6 || || VSD || 7 || || rfib || 7 || || queens || 9 || || rewrite || 9 || || sched || 11 || || fem || 12 || || fibheaps || 13 || || parser || 14 || || rsa || 14 || || genfft || 15 || || clausify || 17 || || gamteb || 17 || || fft || 18 || || boyer || 20 || || gcd || 20 || || sphere || 21 || || fft2 || 23 || || infer || 25 || || maillist || 25 || || mandel || 27 || || nucleic2 || 30 || || primes || 34 || || cichelli || 35 || || ida || 39 || || listcompr || 41 || || listcopy || 43 || || multiplier || 43 || || wang || 43 || || hpg || 45 || || anna || 46 || || reverse-complem || 46 || || integrate || 48 || || primetest || 50 || || paraffins || 54 || || solid || 54 || || puzzle || 55 || || treejoin || 61 || || compress2 || 64 || || compress || 65 || || event || 65 || || bernouilli || 74 || || comp_lab_zift || 78 || || simple || 86 || || wheel-sieve2 || 91 || || life || 100 || || exp3_8 || 102 || || wave4main || 103 || || typecheck || 106 || || atom || 110 || || fulsom || 112 || || CS || 116 || || para || 120 || || kahan || 138 || || transform || 145 || || power || 150 || || cacheprof || 151 || || hidden || 160 || || scs || 168 || || lcss || 175 || || wheel-sieve1 || 199 || || VS || 204 || || pidigits || 216 || || FS || 222 || || last-piece || 230 || || fasta || 263 || || cryptarithm1 || 264 || || CSD || 272 || || digits-of-e1 || 273 || || binary-trees || 342 || || digits-of-e2 || 381 || || circsim || 418 || || S || 451 || || VSM || 497 || || lambda || 617 || || integer || 905 || || n-body || 1043 || || linear || 1070 || || spectral-norm || 1132 || || constraints || 1171 || || exact-reals || 1289 || || mate || 1955 || || fannkuch-redux || 2185 || || k-nucleotide || 2989 || -- Comment (by AndreasK): Formatting fix -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:1> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: (none) Type: task | Status: new Priority: normal | Milestone: 8.6.1 Component: NoFib benchmark | Version: 8.4.3 suite | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by sgraf): * cc: sgraf (added) -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:2> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: (none) Type: task | Status: new Priority: normal | Milestone: 8.6.1 Component: NoFib benchmark | Version: 8.4.3 suite | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): I think that would be extremely helpful. Are you offering to help?! It would be great if so. NB: there's already FAST/NORM/SLOW setting that is supposed to choose per- benchmarks configurations that take 1s/5s/20s or something vaguely like that. Worth figuring out what this does. The configs are doubtless way out of date. Thanks! -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:3> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: (none) Type: task | Status: new Priority: normal | Milestone: 8.6.1 Component: NoFib benchmark | Version: 8.4.3 suite | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by AndreasK):
Are you offering to help?! It would be great if so.
I'm currently working on the NCG for Summer of Code so not before that is over. I might get around to it after, or I might never so if someone else picks this up I would be more than grateful.
there's already FAST/NORM/SLOW setting that is supposed to choose per- benchmarks configurations that take 1s/5s/20s or something vaguely like that.
Makefiles can set different parameters for the different settings. While helpful to main issue is that: * There are a benchmarks which use the same setting for all three variants. * It doesn't help with keeping runtimes of different benchmarks in the same ballpark. Ideally we would try to balance runtime so that for example: * Fast (0.1s-1s) - Enough for allocation/instruction counts or rough runtime measurements. * normal (1s-2s) - Decent runtime measurements * slow (5s-10s) - More precision. Currently runtimes for slow just in shootout vary from 0.5s to 40s. So if we only run these two benchmarks we split benchmark time <2%:>98%. Not exactly a good use of resources. One side effect of adjusting runtimes would be that it would invalidate any long term performance comparison. So I'm also not sure if such a patch would be uncontroversial. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:4> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: (none) Type: task | Status: new Priority: normal | Milestone: 8.6.1 Component: NoFib benchmark | Version: 8.4.3 suite | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by AndreasK): So I also remembered today that nofib-analysis by default ignores results faster than 0.2s for the summary. Which is 91 out of 113 Benchmarks in the above run. I will write up a patch that at least updates speeds for normal mode on the benchmarks where it seems easy/useful to do. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:5> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: (none) Type: task | Status: patch Priority: normal | Milestone: 8.6.1 Component: NoFib benchmark | Version: 8.4.3 suite | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4989 Wiki Page: | -------------------------------------+------------------------------------- Changes (by AndreasK): * status: new => patch * differential: => Phab:D4989 -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:6> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: (none) Type: task | Status: patch Priority: normal | Milestone: 8.6.1 Component: NoFib benchmark | Version: 8.4.3 suite | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #5793 | Differential Rev(s): Phab:D4989 Wiki Page: | -------------------------------------+------------------------------------- Changes (by AndreasK): * related: => #5793 -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:7> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: (none) Type: task | Status: patch Priority: normal | Milestone: 8.8.1 Component: NoFib benchmark | Version: 8.4.3 suite | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #5793 | Differential Rev(s): Phab:D4989 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * milestone: 8.6.1 => 8.8.1 Comment: I'm not sure we want this to be in 8.8. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:8> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: AndreasK Type: task | Status: patch Priority: normal | Milestone: 8.8.1 Component: NoFib benchmark | Version: 8.4.3 suite | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #5793 | Differential Rev(s): Phab:D4989 Wiki Page: | -------------------------------------+------------------------------------- Changes (by AndreasK): * owner: (none) => AndreasK -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:9> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: AndreasK Type: task | Status: patch Priority: normal | Milestone: 8.8.1 Component: NoFib benchmark | Version: 8.4.3 suite | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #5793 #9476 | Differential Rev(s): Phab:D4989 Wiki Page: | -------------------------------------+------------------------------------- Changes (by sgraf): * related: #5793 => #5793 #9476 Comment: C.f. https://ghc.haskell.org/trac/ghc/ticket/9476#comment:52. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:10> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: AndreasK Type: task | Status: patch Priority: normal | Milestone: 8.8.1 Component: NoFib benchmark | Version: 8.4.3 suite | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #5793 #9476 | Differential Rev(s): Phab:D4989 Wiki Page: | -------------------------------------+------------------------------------- Comment (by sgraf): I guess https://ghc.haskell.org/trac/ghc/ticket/5793#comment:38 should have been posted here, as this ticket is clearly more focused on runtime measurements. Just posting this here for reference. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:11> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: (none) Type: task | Status: patch Priority: normal | Milestone: 8.8.1 Component: NoFib benchmark | Version: 8.4.3 suite | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #5793 #9476 | Differential Rev(s): Phab:D4989 Wiki Page: | -------------------------------------+------------------------------------- Changes (by AndreasK): * owner: AndreasK => (none) Comment: I opened this one to track the work I ended up doing in Phab:D4989. Which was essentially an excercise in number tweaking and which I consider done. There is no reason why we can't expand the scope of this ticket by the points raised in your other comment though if you feel they are better suited to be discussed seperately form making nofib awesome. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:12> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: (none) Type: task | Status: closed Priority: normal | Milestone: 8.8.1 Component: NoFib benchmark | Version: 8.4.3 suite | Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #5793 #9476 | Differential Rev(s): Phab:D4989 Wiki Page: | -------------------------------------+------------------------------------- Changes (by sgraf): * status: patch => closed * resolution: => fixed Comment: Right. Let's close this one. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:13> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: (none) Type: task | Status: closed Priority: normal | Milestone: 8.8.1 Component: NoFib benchmark | Version: 8.4.3 suite | Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: #5793 #9476 | Differential Rev(s): Phab:D4989 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Sebastian Graf <sebastian.graf@…>): In [changeset:"8632268ad8405f0c01aaad3ad16e23c65771ba49/nofib" 8632268/nofib]: {{{ #!CommitTicketReference repository="nofib" revision="8632268ad8405f0c01aaad3ad16e23c65771ba49" Stabilise benchmarks wrt. GC Summary: This is due to #15999, a follow-up on #5793 and #15357 and changes all benchmarks, some of them (i.e. `wheel-sieve1`, `awards`) rather drastically. The general plan is outlined in #15999: Identify GC-sensitive benchmarks by looking at how productivity rates change over different nursery sizes and iterate `main` of these benchmarks often enough for the non-monotony and discontinuities to go away. I was paying attention that the benchmarked logic is actually run $n times more often, rather than just benchmarking IO operations printing the result of CAFs. When I found benchmarks with insignificant runtime (#15357), I made sure that parameters/input files were adjusted so that runtime of the different modes fall within the ranges proposed in https://ghc.haskell.org/trac/ghc/ticket/15357#comment:4 - fast: 0.1-0.2s - norm: 1-2s - slow: 5-10s This is what I did: - Stabilise bernoulli - Stabilise digits-of-e1 - Stabilise digits-of-e2 - Stabilise gen_regexp - Adjust running time of integrate - Adjust running time of kahan - Stabilise paraffins - Stabilise primes - Adjust running time of rfib - Adjust running time of tak - Stabilise wheel-sieve1 - Stabilise wheel-sieve2 - Adjust running time of x2n1 - Adjust running time of ansi - Adjust running time of atom - Make awards benchmark something other than IO - Adjust running time of banner - Stabilise boyer - Adjust running time of boyer2 - Adjust running time of queens - Adjust running time of calendar - Adjust runtime of cichelli - Stabilise circsim - Stabilise clausify - Stabilise constraints with moderate success - Adjust running time of cryptarithm1 - Adjust running time of cryptarythm2 - Adjust running time of cse - Adjust running time of eliza - Adjust running time of exact-reals - Adjust running time of expert - Stabilise fft2 - Stabilise fibheaps - Stabilise fish - Adjust running time for gcd - Stabilise comp_lab_zift - Stabilise event - Stabilise fft - Stabilise genfft - Stabilise ida - Adjust running time for listcompr - Adjust running time for listcopy - Adjust running time of nucleic2 - Attempt to stabilise parstof - Stabilise sched - Stabilise solid - Adjust running time of transform - Adjust running time of typecheck - Stabilise wang - Stabilise wave4main - Adjust running time of integer - Adjust running time of knights - Stabilise lambda - Stabilise lcss - Stabilise life - Stabilise mandel - Stabilise mandel2 - Adjust running time of mate - Stabilise minimax - Adjust running time of multiplier - Adjust running time of para - Stabilise power - Adjust running time of primetest - Stabilise puzzle with mild success - Adjust running time for rewrite - Stabilise simple with mild success - Stabilise sorting - Stabilise sphere - Stabilise treejoin - Stabilise anna - Stabilise bspt - Stabilise cacheprof - Stablise compress - Stablise compress2 - Stabilise fem - Adjust running time of fluid - Stabilise fulsom - Stabilise gamteb - Stabilise gg - Stabilise grep - Adjust running time of hidden - Stabilise hpg - Stabilise infer - Stabilise lift - Stabilise linear - Attempt to stabilise maillist - Stabilise mkhprog - Stabilise parser - Stabilise pic - Stabilise prolog - Attempt to stabilise reptile - Adjust running time of rsa - Adjust running time of scs - Stabilise symalg - Stabilise veritas - Stabilise binary-trees - Adjust running time of fasta - Adjust running time of k-nucleotide - Adjust running time of pidigits - Adjust running time of reverse-complement - Adjust running time of spectral-norm - Adjust running time of fannkuch-redux - Adjust running time for n-body Problematic benchmarks: - `last-piece`: Unclear how to stabilise. Runs for 300ms and I can't make up smaller inputs because I don't understand what it does. - `pretty`: It's just much too small to be relevant at all. Maybe we want to get rid of this one? - `scc`: Same as `pretty`. The input graph for which SCC analysis is done is much too small and I can't find good directed example graphs on the internet. - `secretary`: Apparently this needs `-package random` and consequently hasn't been run for a long time. - `simple`: Same as `last-piece`. Decent runtime (70ms), but it's unstable and I see no way to iterate it ~100 times in fast mode. - `eff`: Every benchmark is problematic here. Not from the point of view of allocations, but because the actual logic is vacuous. IMO, these should be performance tests, not actual benchmarks. Alternatively, write an actual application that makes use of algebraic effects. - `maillist`: Too trivial. It's just String/list manipulation, not representative of any Haskell code we would write today (no use of base library functions which could be fused, uses String instead of Text). It's only 75 loc according to `cloc`, that's not a `real` application. Reviewers: simonpj, simonmar, bgamari, AndreasK, osa1, alpmestan, O26 nofib GHC Trac Issues: #15999 Differential Revision: https://phabricator.haskell.org/D5438 }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:14> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
participants (1)
-
GHC