#15357: Make nofib suitable for runtime measurements. -------------------------------------+------------------------------------- Reporter: AndreasK | Owner: (none) Type: task | Status: new Priority: normal | Milestone: 8.6.1 Component: NoFib | Version: 8.4.3 benchmark suite | Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Currently many benchmarks in a default nofib run have runtimes so low that they become meaningless. This comes with a bunch of issues: * It takes up build/runtime while not actually improving the accuracy of runtime measurements. * It skews results as the jump from 1ms to 2ms registers as a 200% change. * It can hide regression which fall outside of the measured granularity. * Higher number of runs per benchmark spend far too much time on the few slower benchmarks. At the extreme end there are 9 benchmarks with runtimes shorter than the granularity of 1ms. For some benchmarks changing this might be as easy as increasing the number of runs/iterations. For others changing problem sizes is not as easy since they process more complex input data. Below are the runtimes of a recent default nofib run. ||= benchmark =|| Time (ms) || || ansi || 0 || || awards || 0 || || banner || 0 || || calendar || 0 || || expert || 0 || || gen_regexps || 0 || || grep || 0 || || pretty || 0 || || scc || 0 || || cse || 1 || || eliza || 1 || || lift || 1 || || mkhprog || 1 || || prolog || 1 || || sorting || 1 || || veritas || 1 || || knights || 2 || || minimax || 2 || || x2n1 || 2 || || mandel2 || 3 || || parstof || 3 || || fluid || 4 || || pic || 4 || || boyer2 || 5 || || bspt || 5 || || cryptarithm2 || 5 || || fish || 6 || || gg || 6 || || reptile || 6 || || symalg || 6 || || tak || 6 || || VSD || 7 || || rfib || 7 || || queens || 9 || || rewrite || 9 || || sched || 11 || || fem || 12 || || fibheaps || 13 || || parser || 14 || || rsa || 14 || || genfft || 15 || || clausify || ansi || 0 || || awards || 0 || || banner || 0 || || calendar || 0 || || expert || 0 || || gen_regexps || 0 || || grep || 0 || || pretty || 0 || || scc || 0 || || cse || 1 || || eliza || 1 || || lift || 1 || || mkhprog || 1 || || prolog || 1 || || sorting || 1 || || veritas || 1 || || knights || 2 || || minimax || 2 || || x2n1 || 2 || || mandel2 || 3 || || parstof || 3 || || fluid || 4 || || pic || 4 || || boyer2 || 5 || || bspt || 5 || || cryptarithm2 || 5 || || fish || 6 || || gg || 6 || || reptile || 6 || || symalg || 6 || || tak || 6 || || VSD || 7 || || rfib || 7 || || queens || 9 || || rewrite || 9 || || sched || 11 || || fem || 12 || || fibheaps || 13 || || parser || 14 || || rsa || 14 || || genfft || 15 || || clausify || 17 || || gamteb || 17 || || fft || 18 || || boyer || 20 || || gcd || 20 || || sphere || 21 || || fft2 || 23 || || infer || 25 || || maillist || 25 || || mandel || 27 || || nucleic2 || 30 || || primes || 34 || || cichelli || 35 || || ida || 39 || || listcompr || 41 || || listcopy || 43 || || multiplier || 43 || || wang || 43 || || hpg || 45 || || anna || 46 || || reverse-complem || 46 || || integrate || 48 || || primetest || 50 || || paraffins || 54 || || solid || 54 || || puzzle || 55 || || treejoin || 61 || || compress2 || 64 || || compress || 65 || || event || 65 || || bernouilli || 74 || || comp_lab_zift || 78 || || simple || 86 || || wheel-sieve2 || 91 || || life || 100 || || exp3_8 || 102 || || wave4main || 103 || || typecheck || 106 || || atom || 110 || || fulsom || 112 || || CS || 116 || || para || 120 || || kahan || 138 || || transform || 145 || || power || 150 || || cacheprof || 151 || || hidden || 160 || || scs || 168 || || lcss || 175 || || wheel-sieve1 || 199 || || VS || 204 || || pidigits || 216 || || FS || 222 || || last-piece || 230 || || fasta || 263 || || cryptarithm1 || 264 || || CSD || 272 || || digits-of-e1 || 273 || || binary-trees || 342 || || digits-of-e2 || 381 || || circsim || 418 || || S || 451 || || VSM || 497 || || lambda || 617 || || integer || 905 || || n-body || 1043 || || linear || 1070 || || spectral-norm || 1132 || || constraints || 1171 || || exact-reals || 1289 || || mate || 1955 || || fannkuch-redux || 2185 || || k-nucleotide || 2989 || || 17 || || gamteb || 17 || || fft || 18 || || boyer || 20 || || gcd || 20 || || sphere || 21 || || fft2 || 23 || || infer || 25 || || maillist || 25 || || mandel || 27 || || nucleic2 || 30 || || primes || 34 || || cichelli || 35 || || ida || 39 || || listcompr || 41 || || listcopy || 43 || || multiplier || 43 || || wang || 43 || || hpg || 45 || || anna || 46 || || reverse-complem || 46 || || integrate || 48 || || primetest || 50 || || paraffins || 54 || || solid || 54 || || puzzle || 55 || || treejoin || 61 || || compress2 || 64 || || compress || 65 || || event || 65 || || bernouilli || 74 || || comp_lab_zift || 78 || || simple || 86 || || wheel-sieve2 || 91 || || life || 100 || || exp3_8 || 102 || || wave4main || 103 || || typecheck || 106 || || atom || 110 || || fulsom || 112 || || CS || 116 || || para || 120 || || kahan || 138 || || transform || 145 || || power || 150 || || cacheprof || 151 || || hidden || 160 || || scs || 168 || || lcss || 175 || || wheel-sieve1 || 199 || || VS || 204 || || pidigits || 216 || || FS || 222 || || last-piece || 230 || || fasta || 263 || || cryptarithm1 || 264 || || CSD || 272 || || digits-of-e1 || 273 || || binary-trees || 342 || || digits-of-e2 || 381 || || circsim || 418 || || S || 451 || || VSM || 497 || || lambda || 617 || || integer || 905 || || n-body || 1043 || || linear || 1070 || || spectral-norm || 1132 || || constraints || 1171 || || exact-reals || 1289 || || mate || 1955 || || fannkuch-redux || 2185 || || k-nucleotide || 2989 || -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15357 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler