Re: [GHC] #15357: Make nofib suitable for runtime measurements.

9 Jul 2018

      #15357: Make nofib suitable for runtime measurements.
-------------------------------------+-------------------------------------
        Reporter:  AndreasK          |                Owner:  (none)
            Type:  task              |               Status:  new
        Priority:  normal            |            Milestone:  8.6.1
       Component:  NoFib benchmark   |              Version:  8.4.3
  suite                              |
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Old description:
...
Currently many benchmarks in a default nofib run have runtimes so low
that they become meaningless.
This comes with a bunch of issues:
* It takes up build/runtime while not actually improving the accuracy of
runtime measurements.
* It skews results as the jump from 1ms to 2ms registers as a 200%
change.
* It can hide regression which fall outside of the measured granularity.
* Higher number of runs per benchmark spend far too much time on the few
slower benchmarks.
At the extreme end there are 9 benchmarks with runtimes shorter than the
granularity of 1ms.
For some benchmarks changing this might be as easy as increasing the
number of runs/iterations.
For others changing problem sizes is not as easy since they process more
complex input data.
Below are the runtimes of a recent default nofib run.

...
||= benchmark =|| Time (ms) ||
||      ansi    ||      0       ||
||      awards  ||      0       ||
||      banner  ||      0       ||
||      calendar        ||      0       ||
||      expert  ||      0       ||
||      gen_regexps     ||      0       ||
||      grep    ||      0       ||
||      pretty  ||      0       ||
||      scc     ||      0       ||
||      cse     ||      1       ||
||      eliza   ||      1       ||
||      lift    ||      1       ||
||      mkhprog ||      1       ||
||      prolog  ||      1       ||
||      sorting ||      1       ||
||      veritas ||      1       ||
||      knights ||      2       ||
||      minimax ||      2       ||
||      x2n1    ||      2       ||
||      mandel2 ||      3       ||
||      parstof ||      3       ||
||      fluid   ||      4       ||
||      pic     ||      4       ||
||      boyer2  ||      5       ||
||      bspt    ||      5       ||
||      cryptarithm2    ||      5       ||
||      fish    ||      6       ||
||      gg      ||      6       ||
||      reptile ||      6       ||
||      symalg  ||      6       ||
||      tak     ||      6       ||
||      VSD     ||      7       ||
||      rfib    ||      7       ||
||      queens  ||      9       ||
||      rewrite ||      9       ||
||      sched   ||      11      ||
||      fem     ||      12      ||
||      fibheaps        ||      13      ||
||      parser  ||      14      ||
||      rsa     ||      14      ||
||      genfft  ||      15      ||
||      clausify        ||      ansi    ||      0       ||
||      awards  ||      0       ||
||      banner  ||      0       ||
||      calendar        ||      0       ||
||      expert  ||      0       ||
||      gen_regexps     ||      0       ||
||      grep    ||      0       ||
||      pretty  ||      0       ||
||      scc     ||      0       ||
||      cse     ||      1       ||
||      eliza   ||      1       ||
||      lift    ||      1       ||
||      mkhprog ||      1       ||
||      prolog  ||      1       ||
||      sorting ||      1       ||
||      veritas ||      1       ||
||      knights ||      2       ||
||      minimax ||      2       ||
||      x2n1    ||      2       ||
||      mandel2 ||      3       ||
||      parstof ||      3       ||
||      fluid   ||      4       ||
||      pic     ||      4       ||
||      boyer2  ||      5       ||
||      bspt    ||      5       ||
||      cryptarithm2    ||      5       ||
||      fish    ||      6       ||
||      gg      ||      6       ||
||      reptile ||      6       ||
||      symalg  ||      6       ||
||      tak     ||      6       ||
||      VSD     ||      7       ||
||      rfib    ||      7       ||
||      queens  ||      9       ||
||      rewrite ||      9       ||
||      sched   ||      11      ||
||      fem     ||      12      ||
||      fibheaps        ||      13      ||
||      parser  ||      14      ||
||      rsa     ||      14      ||
||      genfft  ||      15      ||
||      clausify        ||      17      ||
||      gamteb  ||      17      ||
||      fft     ||      18      ||
||      boyer   ||      20      ||
||      gcd     ||      20      ||
||      sphere  ||      21      ||
||      fft2    ||      23      ||
||      infer   ||      25      ||
||      maillist        ||      25      ||
||      mandel  ||      27      ||
||      nucleic2        ||      30      ||
||      primes  ||      34      ||
||      cichelli        ||      35      ||
||      ida     ||      39      ||
||      listcompr       ||      41      ||
||      listcopy        ||      43      ||
||      multiplier      ||      43      ||
||      wang    ||      43      ||
||      hpg     ||      45      ||
||      anna    ||      46      ||
||      reverse-complem ||      46      ||
||      integrate       ||      48      ||
||      primetest       ||      50      ||
||      paraffins       ||      54      ||
||      solid   ||      54      ||
||      puzzle  ||      55      ||
||      treejoin        ||      61      ||
||      compress2       ||      64      ||
||      compress        ||      65      ||
||      event   ||      65      ||
||      bernouilli      ||      74      ||
||      comp_lab_zift   ||      78      ||
||      simple  ||      86      ||
||      wheel-sieve2    ||      91      ||
||      life    ||      100     ||
||      exp3_8  ||      102     ||
||      wave4main       ||      103     ||
||      typecheck       ||      106     ||
||      atom    ||      110     ||
||      fulsom  ||      112     ||
||      CS      ||      116     ||
||      para    ||      120     ||
||      kahan   ||      138     ||
||      transform       ||      145     ||
||      power   ||      150     ||
||      cacheprof       ||      151     ||
||      hidden  ||      160     ||
||      scs     ||      168     ||
||      lcss    ||      175     ||
||      wheel-sieve1    ||      199     ||
||      VS      ||      204     ||
||      pidigits        ||      216     ||
||      FS      ||      222     ||
||      last-piece      ||      230     ||
||      fasta   ||      263     ||
||      cryptarithm1    ||      264     ||
||      CSD     ||      272     ||
||      digits-of-e1    ||      273     ||
||      binary-trees    ||      342     ||
||      digits-of-e2    ||      381     ||
||      circsim ||      418     ||
||      S       ||      451     ||
||      VSM     ||      497     ||
||      lambda  ||      617     ||
||      integer ||      905     ||
||      n-body  ||      1043    ||
||      linear  ||      1070    ||
||      spectral-norm   ||      1132    ||
||      constraints     ||      1171    ||
||      exact-reals     ||      1289    ||
||      mate    ||      1955    ||
||      fannkuch-redux  ||      2185    ||
||      k-nucleotide    ||      2989    ||
||      17      ||
||      gamteb  ||      17      ||
||      fft     ||      18      ||
||      boyer   ||      20      ||
||      gcd     ||      20      ||
||      sphere  ||      21      ||
||      fft2    ||      23      ||
||      infer   ||      25      ||
||      maillist        ||      25      ||
||      mandel  ||      27      ||
||      nucleic2        ||      30      ||
||      primes  ||      34      ||
||      cichelli        ||      35      ||
||      ida     ||      39      ||
||      listcompr       ||      41      ||
||      listcopy        ||      43      ||
||      multiplier      ||      43      ||
||      wang    ||      43      ||
||      hpg     ||      45      ||
||      anna    ||      46      ||
||      reverse-complem ||      46      ||
||      integrate       ||      48      ||
||      primetest       ||      50      ||
||      paraffins       ||      54      ||
||      solid   ||      54      ||
||      puzzle  ||      55      ||
||      treejoin        ||      61      ||
||      compress2       ||      64      ||
||      compress        ||      65      ||
||      event   ||      65      ||
||      bernouilli      ||      74      ||
||      comp_lab_zift   ||      78      ||
||      simple  ||      86      ||
||      wheel-sieve2    ||      91      ||
||      life    ||      100     ||
||      exp3_8  ||      102     ||
||      wave4main       ||      103     ||
||      typecheck       ||      106     ||
||      atom    ||      110     ||
||      fulsom  ||      112     ||
||      CS      ||      116     ||
||      para    ||      120     ||
||      kahan   ||      138     ||
||      transform       ||      145     ||
||      power   ||      150     ||
||      cacheprof       ||      151     ||
||      hidden  ||      160     ||
||      scs     ||      168     ||
||      lcss    ||      175     ||
||      wheel-sieve1    ||      199     ||
||      VS      ||      204     ||
||      pidigits        ||      216     ||
||      FS      ||      222     ||
||      last-piece      ||      230     ||
||      fasta   ||      263     ||
||      cryptarithm1    ||      264     ||
||      CSD     ||      272     ||
||      digits-of-e1    ||      273     ||
||      binary-trees    ||      342     ||
||      digits-of-e2    ||      381     ||
||      circsim ||      418     ||
||      S       ||      451     ||
||      VSM     ||      497     ||
||      lambda  ||      617     ||
||      integer ||      905     ||
||      n-body  ||      1043    ||
||      linear  ||      1070    ||
||      spectral-norm   ||      1132    ||
||      constraints     ||      1171    ||
||      exact-reals     ||      1289    ||
||      mate    ||      1955    ||
||      fannkuch-redux  ||      2185    ||
||      k-nucleotide    ||      2989    ||
New description:

 Currently many benchmarks in a default nofib run have runtimes so low that
 they become meaningless.

 This comes with a bunch of issues:
 * It takes up build/runtime while not actually improving the accuracy of
 runtime measurements.
 * It skews results as the jump from 1ms to 2ms registers as a 200% change.
 * It can hide regression which fall outside of the measured granularity.
 * Higher number of runs per benchmark spend far too much time on the few
 slower benchmarks.

 At the extreme end there are 9 benchmarks with runtimes shorter than the
 granularity of 1ms.

 For some benchmarks changing this might be as easy as increasing the
 number of runs/iterations.
 For others changing problem sizes is not as easy since they process more
 complex input data.

 Below are the runtimes of a recent default nofib run.

 ||= benchmark =|| Time (ms) ||
 ||      ansi    ||      0       ||
 ||      awards  ||      0       ||
 ||      banner  ||      0       ||
 ||      calendar        ||      0       ||
 ||      expert  ||      0       ||
 ||      gen_regexps     ||      0       ||
 ||      grep    ||      0       ||
 ||      pretty  ||      0       ||
 ||      scc     ||      0       ||
 ||      cse     ||      1       ||
 ||      eliza   ||      1       ||
 ||      lift    ||      1       ||
 ||      mkhprog ||      1       ||
 ||      prolog  ||      1       ||
 ||      sorting ||      1       ||
 ||      veritas ||      1       ||
 ||      knights ||      2       ||
 ||      minimax ||      2       ||
 ||      x2n1    ||      2       ||
 ||      mandel2 ||      3       ||
 ||      parstof ||      3       ||
 ||      fluid   ||      4       ||
 ||      pic     ||      4       ||
 ||      boyer2  ||      5       ||
 ||      bspt    ||      5       ||
 ||      cryptarithm2    ||      5       ||
 ||      fish    ||      6       ||
 ||      gg      ||      6       ||
 ||      reptile ||      6       ||
 ||      symalg  ||      6       ||
 ||      tak     ||      6       ||
 ||      VSD     ||      7       ||
 ||      rfib    ||      7       ||
 ||      queens  ||      9       ||
 ||      rewrite ||      9       ||
 ||      sched   ||      11      ||
 ||      fem     ||      12      ||
 ||      fibheaps        ||      13      ||
 ||      parser  ||      14      ||
 ||      rsa     ||      14      ||
 ||      genfft  ||      15      ||
 ||      clausify        ||      17      ||
 ||      gamteb  ||      17      ||
 ||      fft     ||      18      ||
 ||      boyer   ||      20      ||
 ||      gcd     ||      20      ||
 ||      sphere  ||      21      ||
 ||      fft2    ||      23      ||
 ||      infer   ||      25      ||
 ||      maillist        ||      25      ||
 ||      mandel  ||      27      ||
 ||      nucleic2        ||      30      ||
 ||      primes  ||      34      ||
 ||      cichelli        ||      35      ||
 ||      ida     ||      39      ||
 ||      listcompr       ||      41      ||
 ||      listcopy        ||      43      ||
 ||      multiplier      ||      43      ||
 ||      wang    ||      43      ||
 ||      hpg     ||      45      ||
 ||      anna    ||      46      ||
 ||      reverse-complem ||      46      ||
 ||      integrate       ||      48      ||
 ||      primetest       ||      50      ||
 ||      paraffins       ||      54      ||
 ||      solid   ||      54      ||
 ||      puzzle  ||      55      ||
 ||      treejoin        ||      61      ||
 ||      compress2       ||      64      ||
 ||      compress        ||      65      ||
 ||      event   ||      65      ||
 ||      bernouilli      ||      74      ||
 ||      comp_lab_zift   ||      78      ||
 ||      simple  ||      86      ||
 ||      wheel-sieve2    ||      91      ||
 ||      life    ||      100     ||
 ||      exp3_8  ||      102     ||
 ||      wave4main       ||      103     ||
 ||      typecheck       ||      106     ||
 ||      atom    ||      110     ||
 ||      fulsom  ||      112     ||
 ||      CS      ||      116     ||
 ||      para    ||      120     ||
 ||      kahan   ||      138     ||
 ||      transform       ||      145     ||
 ||      power   ||      150     ||
 ||      cacheprof       ||      151     ||
 ||      hidden  ||      160     ||
 ||      scs     ||      168     ||
 ||      lcss    ||      175     ||
 ||      wheel-sieve1    ||      199     ||
 ||      VS      ||      204     ||
 ||      pidigits        ||      216     ||
 ||      FS      ||      222     ||
 ||      last-piece      ||      230     ||
 ||      fasta   ||      263     ||
 ||      cryptarithm1    ||      264     ||
 ||      CSD     ||      272     ||
 ||      digits-of-e1    ||      273     ||
 ||      binary-trees    ||      342     ||
 ||      digits-of-e2    ||      381     ||
 ||      circsim ||      418     ||
 ||      S       ||      451     ||
 ||      VSM     ||      497     ||
 ||      lambda  ||      617     ||
 ||      integer ||      905     ||
 ||      n-body  ||      1043    ||
 ||      linear  ||      1070    ||
 ||      spectral-norm   ||      1132    ||
 ||      constraints     ||      1171    ||
 ||      exact-reals     ||      1289    ||
 ||      mate    ||      1955    ||
 ||      fannkuch-redux  ||      2185    ||
 ||      k-nucleotide    ||      2989    ||

--

Comment (by AndreasK):

 Formatting fix

-- 
Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/15357#comment:1
GHC http://www.haskell.org/ghc/
The Glasgow Haskell Compiler