You should be able to use Laborantin to organize this kind of experiments. You'll have to write some boilerplate on how to "build" each package set.2015-06-11 12:45 GMT+01:00 Alp Mestanogullari <alpmestan@gmail.com>:_______________________________________________Oh, Julian Arni just reminded me of Joachim Breitner's Gipeda: https://github.com/nomeata/gipedaThis might do the trick, but any suggestion is still welcome!--On Thu, Jun 11, 2015 at 1:41 PM, Alp Mestanogullari <alpmestan@gmail.com> wrote:Hi -cafe,While we can easily benchmark different functions or libraries easily with criterion, I can't think of a reasonably easy (and accurate!) way of benchmarking two versions of the same package. And not necessarily version as in cabal version -- one of the use cases I have in mind would be running a benchmark suite whenever a PR gets merged to the main branch of a library, so the benchmark would need to comparethe performance of the library's-code-before-merging and after.This definitely can't be accomplished with something like criterion because we can't have two different instances of a package in scope for a module, even with -XPackageImports.If we separately build the same program against two instances of the same library and run the benchmarks separately, this might happen far apart enough that the machine running this might be under a different load. This does however seem to be the only actual solution? Run separately at two different commits, diff the numbers, report.I have felt the need for a solution to this for quite some time, but have never needed it bad enough that it became a priority. Thinking about it again today, I thought I should drop an email to the list and see if fellow haskellers have an easy-to-use solution for this? I may very well be overlooking something.--Alp MestanogullariAlp Mestanogullari
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe