Hello Johan, Off the top of my head: Technically speaking, the compiler does not have to be optimized to do build tests; just the libraries. So you could probably make the builds run a bit faster this way. Your command for nofib will only run the "standard" benchmarks; to run, for example, the GC benchmarks, you will have to cd into the gc directory and run make there. You may also need to vary the tests parameters in other ways, such as threaded and un-threaded. Cheers, Edward Excerpts from Johan Tibell's message of 2014-03-06 01:50:09 -0800:

Hi,

I'd like to set up a performance build bot for GHC, but before I can do that I need a script that reliably builds GHC and runs nofib. Do we have such a script? Here's a strawman proposal for one:

cabal install happy alex git clone git://git.haskell.org/ghc.git cd ghc ./sync-all --nofib get perl boot ./configure make cd nofib make clean make boot make -k mode=slow

Questions:

* Does this look sensible? * Is there a way to only build and run a subset of the benchmarks? * Are there any tweaks to mk/build.mk we can do to make the build faster without compromising the results? * Is there a way to do this in a cheap throwaway VM like travis-ci does? Could such a VM already provide GHC and the required libs to make the whole thing hermetic?

On 2014-03-06 at 10:50:09 +0100, Johan Tibell wrote:

I'd like to set up a performance build bot for GHC, but before I can do that I need a script that reliably builds GHC and runs nofib. Do we have such a script? Here's a strawman proposal for one:

cabal install happy alex git clone git://git.haskell.org/ghc.git cd ghc ./sync-all --nofib get perl boot ./configure make cd nofib make clean make boot make -k mode=slow

Questions:

* Does this look sensible?

If the script is supposed to start in an empty environment, then it looks good to me

* Is there a way to only build and run a subset of the benchmarks? * Are there any tweaks to mk/build.mk we can do to make the build faster without compromising the results?

You should disable LaTeX generation, and maybe also haddock generation; and possibly remove the libraries/dph folder (unless that's becomes part of the benchmarks) how often would you run the benchmarks? once a day?

* Is there a way to do this in a cheap throwaway VM like travis-ci does? Could such a VM already provide GHC and the required libs to make the whole thing hermetic?

Btw, does benchmarking inside a VM provide good enough measurements? E.g. I'd be worried from noise due to vSMP scheduling (as GHC's GC is rather sensible to proper SMP scheduling) and maybe from noise coming from other VMs trashing the caches.

On 06/03/14 09:50, Johan Tibell wrote:

Hi,

I'd like to set up a performance build bot for GHC, but before I can do that I need a script that reliably builds GHC and runs nofib. Do we have such a script? Here's a strawman proposal for one:

cabal install happy alex git clone git://git.haskell.org/ghc.git http://git.haskell.org/ghc.git cd ghc ./sync-all --nofib get perl boot

./boot

./configure make cd nofib make clean make boot make -k mode=slow

Questions:

* Does this look sensible? * Is there a way to only build and run a subset of the benchmarks? * Are there any tweaks to mk/build.mk http://build.mk we can do to make the build faster without compromising the results?

Turn down the stage2 optimisation, and turn off the docs: GhcStage2HcOpts = -O HADDOCK_DOCS = NO BUILD_DOCBOOK_HTML = NO BUILD_DOCBOOK_PS = NO BUILD_DOCBOOK_PDF = NO

* Is there a way to do this in a cheap throwaway VM like travis-ci does? Could such a VM already provide GHC and the required libs to make the whole thing hermetic?

Sure. I occasionally use EC2 for GHC hacking and I had a VM set up with all the tools ready to work on GHC. (but I assume you mean a local VM; EC2 is not good for perf measurements) Cheers, Simon