Yeah, although I think this doesn't really solve the issue of using two versions of the same package. On Thu, Jun 11, 2015 at 1:52 PM, lucas di cioccio

wrote:

You should be able to use Laborantin to organize this kind of experiments. You'll have to write some boilerplate on how to "build" each package set. https://hackage.haskell.org/package/laborantin-hs

--Lucas

2015-06-11 12:45 GMT+01:00 Alp Mestanogullari :

...

Oh, Julian Arni just reminded me of Joachim Breitner's Gipeda: https://github.com/nomeata/gipeda

This might do the trick, but any suggestion is still welcome!

On Thu, Jun 11, 2015 at 1:41 PM, Alp Mestanogullari wrote:

...

Hi -cafe,

While we can easily benchmark different functions or libraries easily with criterion, I can't think of a reasonably easy (and accurate!) way of benchmarking two versions of the same package. And not necessarily version as in cabal version -- one of the use cases I have in mind would be running a benchmark suite whenever a PR gets merged to the main branch of a library, so the benchmark would need to compare the performance of the library's-code-before-merging and after.

This definitely can't be accomplished with something like criterion because we can't have two different instances of a package in scope for a module, even with -XPackageImports.

If we separately build the same program against two instances of the same library and run the benchmarks separately, this might happen far apart enough that the machine running this might be under a different load. This does however seem to be the only actual solution? Run separately at two different commits, diff the numbers, report.

I have felt the need for a solution to this for quite some time, but have never needed it bad enough that it became a priority. Thinking about it again today, I thought I should drop an email to the list and see if fellow haskellers have an easy-to-use solution for this? I may very well be overlooking something.

-- Alp Mestanogullari

-- Alp Mestanogullari

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

-- Alp Mestanogullari

I thought about this but discarded it quite early on for the hacky nature. But that definitely looks like my best shot, the more I think about it. Thanks. On Thu, Jun 11, 2015 at 1:48 PM, Roman Cheplyaka wrote:

In your CI script, you can rename one of your versions (say, mypkg to mypkg-new) and then use -XPackageImports.

On 11/06/15 14:41, Alp Mestanogullari wrote:

...

Hi -cafe,

While we can easily benchmark different functions or libraries easily with criterion, I can't think of a reasonably easy (and accurate!) way of benchmarking two versions of the same package. And not necessarily version as in cabal version -- one of the use cases I have in mind would be running a benchmark suite whenever a PR gets merged to the main branch of a library, so the benchmark would need to compare the performance of the library's-code-before-merging and after.

This definitely can't be accomplished with something like criterion because we can't have two different instances of a package in scope for a module, even with -XPackageImports.

If we separately build the same program against two instances of the same library and run the benchmarks separately, this might happen far apart enough that the machine running this might be under a different load. This does however seem to be the only actual solution? Run separately at two different commits, diff the numbers, report.

I have felt the need for a solution to this for quite some time, but have never needed it bad enough that it became a priority. Thinking about it again today, I thought I should drop an email to the list and see if fellow haskellers have an easy-to-use solution for this? I may very well be overlooking something.

-- Alp Mestanogullari

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe

-- Alp Mestanogullari