30 Nov
2012
30 Nov
'12
11:20 a.m.
On Fri, 2012-11-30 at 15:51 +0000, Tim Watson wrote:
Could we not configure travis-ci to run the benchmarks for us or something like that? A simple (free) ci setup would be easier than finding a pair of hands to do this regularly I would've thought.
AFAIK Travis uses some IAAS service (EC2 if I'm not mistaken) to execute CI jobs. Experience has shown benchmark results from VMs (especially running on some public/shared IAAS service) are rather useless: there's huge variance between results, even when executing the exact same binaries, depending on host CPU and IO load (the latter even in case your benchmark doesn't perform any IO itself), time of day (which influences load), actual hardware your VM gets deployed on,... Just my .02, Nicolas