[GHC] #9571: nofib should use criterion-style bootstrapping/sampling
#9571: nofib should use criterion-style bootstrapping/sampling -------------------------------------+------------------------------------- Reporter: ezyang | Owner: Type: feature request | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.9 suite | Operating System: Keywords: | Unknown/Multiple Architecture: Unknown/Multiple | Type of failure: Difficulty: Unknown | None/Unknown Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: -------------------------------------+------------------------------------- As I discovered when investigating situations like #9570, in some cases, test-cases in nofib are giving nonsense, and it's hard to tell unless you run nofib several times and notice that percentage differences are fluctuating up and down. The quality of the numbers we get for uninformed users would be better if we ran some statistical analysis to tell how many times to run the benchmark, and if there were lots of outliers (rather than just blindly summarizing all the runs using an average.) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9571 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9571: nofib should use criterion-style bootstrapping/sampling -------------------------------------+------------------------------------- Reporter: ezyang | Owner: Type: feature | Status: new request | Milestone: Priority: normal | Version: 7.9 Component: NoFib | Keywords: benchmark suite | Architecture: Unknown/Multiple Resolution: | Difficulty: Unknown Operating System: | Blocked By: Unknown/Multiple | Related Tickets: Type of failure: | None/Unknown | Test Case: | Blocking: 5793 | Differential Revisions: | -------------------------------------+------------------------------------- Changes (by ezyang): * blocking: => 5793 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9571#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9571: nofib should use criterion-style bootstrapping/sampling -------------------------------------+------------------------------------- Reporter: ezyang | Owner: Type: feature request | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.9 suite | Keywords: Resolution: | Architecture: Operating System: Unknown/Multiple | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: 5793 Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by kanetw): Is it possible to just use criterion to run the benchmarks, or does GHC have special requirements that would require an own implementation? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9571#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9571: nofib should use criterion-style bootstrapping/sampling -------------------------------------+------------------------------------- Reporter: ezyang | Owner: Type: feature request | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.9 suite | Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: 5793 Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by rwbarton): Replying to [comment:2 kanetw]:
Is it possible to just use criterion to run the benchmarks, or does GHC have special requirements that would require an own implementation?
This is discussed a bit in the comments on #5793. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9571#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC