Hello! I want to announce new benchmarking framework tasty-papi[1,2] Unlike criterion, gauge or tasty-bench is uses instruction counting instead of time measurements. It should be affected by OS scheduler much less so it should be possible to run benchmarks on CI. Although not on github CI (not enough permissions) For measurements papi[3] is used. So all hardware and OS limitations are inherited from there. At least it works on linux Here is example output:
6: OK (0.02s) ALLOC=528 TOT_INS=3609 TOT_CYC=4981 BR_INS=907 BR_MSP=58 7: OK ALLOC=864 TOT_INS=5518 TOT_CYC=5138 BR_INS=1378 BR_MSP=50 8: OK ALLOC=1408 TOT_INS=8614 TOT_CYC=6515 BR_INS=2142 BR_MSP=63
For each benchmark we measure number of allocated bytes, number of instructions and cycles spent as well as number of branching instructions and branch mispredictions. Counter set could be adjusted using command line flags. This library hasn't seen much real world use. So treat measurement with care. [1] https://hackage.haskell.org/package/tasty-papi [2] https://github.com/Shimuuar/tasty-papi [3] https://icl.utk.edu/papi/ - Alexey