#9571: nofib should use criterion-style bootstrapping/sampling -------------------------------------+------------------------------------- Reporter: ezyang | Owner: Type: feature request | Status: new Priority: normal | Milestone: Component: NoFib benchmark | Version: 7.9 suite | Operating System: Keywords: | Unknown/Multiple Architecture: Unknown/Multiple | Type of failure: Difficulty: Unknown | None/Unknown Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: -------------------------------------+------------------------------------- As I discovered when investigating situations like #9570, in some cases, test-cases in nofib are giving nonsense, and it's hard to tell unless you run nofib several times and notice that percentage differences are fluctuating up and down. The quality of the numbers we get for uninformed users would be better if we ran some statistical analysis to tell how many times to run the benchmark, and if there were lots of outliers (rather than just blindly summarizing all the runs using an average.) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9571 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler