Hi, Is CircleCI the future now? In general, yes. But it’s running fine for now, so I would not prematurely throw it over. My requirements are: * needs to run on every commit (not just every push), including branches. * needs to be able to push to a repository¹, so it needs access to some secret token for depolyment. Alternatively, the CI could simply keep track of the build log and the perf.haskell.org dashboard scrapes it from there. * Occasionally, I find that I want to rebuild a fair number of commits that are already in the repository. So a good way to trigger rebuilds would be nice. Greetings, Joachim ¹ https://github.com/nomeata/ghc-speed-logs/commits/master Am Dienstag, den 24.10.2017, 12:18 +1100 schrieb Manuel M T Chakravarty:

Hi Joachim,

Great! Just because you mention CI infrastructure and our effort around GHC CI at the moment, do you think, it would make sense to move this to CircleCI eventually?

Cheers, Manuel

...

Am 24.10.2017 um 11:02 schrieb Joachim Breitner :

Hi,

after a system upgrade to avoid weird linker errors, and after some fixes in the nofib submodule, http://perf.haskell.org/ghc is running again.

I am collecting instruction counts instead of runtime, because the latter was just too often varying too wildly. I hope this will yield less false alarms.

I am also running nofib with mode=fast. This way, building GHC, running the testsuite and nofib takes a bit over one hour. I hope this can keep up with y'all's commits (when it took 2h it couldn't).

Now nothing of the setup requires a quiet dedicated machine, so if there is a need, we could move these builds into the cloud or into some CI infrastructure - but no changes are immediately planned.

Enjoy, Joachim -- Joachim Breitner mail@joachim-breitner.de http://www.joachim-breitner.de/

Hi, Am Dienstag, den 24.10.2017, 09:57 +0200 schrieb Boespflug, Mathieu:

what kind of machine is this running on at the moment? A dedicated instance or some colocated VM in the cloud?

A physical box sponsored by Bryn Mawr. 8GB RAM, 8 Core Intel(R) Core(TM) i7 CPU 930 @ 2.80GHz

I ask because for perf results something CircleCI might *not* be suitable here because execution time on CircleCI varies quite substantially depending on load from other users of the same physical machine. This is not a problem for most types of jobs, but perf measurement is one of the few for which it is, depending on which metrics are tracked.

But that’s what I am saying: By measuring dynamic instructions (using cachegrind), the execution time is no longer relevant. Joachim -- Joachim Breitner mail@joachim-breitner.de http://www.joachim-breitner.de/

Hi, Am Dienstag, den 24.10.2017, 16:59 +0200 schrieb Boespflug, Mathieu:

Hi Joachim,

...

But that’s what I am saying: By measuring dynamic instructions (using cachegrind), the execution time is no longer relevant.

I'm skeptical that this metric is a reliable enough proxy for actual runtime of user programs to make tracking it and only it sufficient. For the reasons Sven Panne gave in a previous email. But I'll be very interested to hear your experience report over time.

Sure, it’s a crutch. But we have been working for decades with “allocations go down, so this must be a win”, which is even more distant from runtime. If someone builds me a system where the runtime measurements are actually reliable (and not drowned in the noise of modern CPU architectures with sensitivity to layout and alignment), maybe using a tool like this http://plasma.cs.umass.edu/emery/stabilizer.html then I’ll happily ditch instruction counts and use that! But at the current state, working with nofib results is just too frustrating: You see a change, but you don't know, is it a measurement error? Did you push something over a performance cliff? Is it really related to your change to Core? I find it just not viable. Joachim -- Joachim Breitner mail@joachim-breitner.de http://www.joachim-breitner.de/

Hi, just wanted to throw in the idea of parallelising the benchmark suite (hurts to even write that, but cachegrind) to speed up the build, if ever need be. Cheers, Sebastian On Wed, Oct 25, 2017 at 2:13 AM, Manuel M T Chakravarty < chak@justtesting.org> wrote:

Am 24.10.2017 um 14:46 schrieb Joachim Breitner

...

:

Is CircleCI the future now?

Yes, as per

https://ghc.haskell.org/trac/ghc/wiki/ContinuousIntegration

In general, yes. But it’s running fine for now, so I would not prematurely throw it over.

My requirements are:

* needs to run on every commit (not just every push), including branches. * needs to be able to push to a repository¹, so it needs access to some secret token for depolyment. Alternatively, the CI could simply keep track of the build log and the perf.haskell.org dashboard scrapes it from there. * Occasionally, I find that I want to rebuild a fair number of commits that are already in the repository. So a good way to trigger rebuilds would be nice.

No need to mess with a working system, but as long as performance counters are the basis, I think, these constraints can all be met. In particular, CircleCI has facilities to allow deployments including managing of secrets. (That is a pretty common requirements for CIs.)

Cheers, Manuel

Greetings, Joachim

¹ https://github.com/nomeata/ghc-speed-logs/commits/master

Am Dienstag, den 24.10.2017, 12:18 +1100 schrieb Manuel M T Chakravarty:

Hi Joachim,

Great! Just because you mention CI infrastructure and our effort around GHC CI at the moment, do you think, it would make sense to move this to CircleCI eventually?

Cheers, Manuel

Am 24.10.2017 um 11:02 schrieb Joachim Breitner

...

:

Hi,

after a system upgrade to avoid weird linker errors, and after some fixes in the nofib submodule, http://perf.haskell.org/ghc is running again.

I am collecting instruction counts instead of runtime, because the latter was just too often varying too wildly. I hope this will yield less false alarms.

I am also running nofib with mode=fast. This way, building GHC, running the testsuite and nofib takes a bit over one hour. I hope this can keep up with y'all's commits (when it took 2h it couldn't).

Now nothing of the setup requires a quiet dedicated machine, so if there is a need, we could move these builds into the cloud or into some CI infrastructure - but no changes are immediately planned.

Enjoy, Joachim

-- Joachim Breitner mail@joachim-breitner.de http://www.joachim-breitner.de/ _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Hi, just came here to resurrect the thread and point out that nofib currently isn't really run in parallel: https://ghc.haskell.org/trac/ghc/ticket/15976#ticket Also it's unclear to me how that would be possible without rewriting the whole benchmark harness in Python or Haskell, because make parallelism would probably interleave outputs of different rules, giving nofib-analyse a hard time. Greetings Sebastian Am Mo., 30. Okt. 2017 um 00:14 Uhr schrieb Joachim Breitner < mail@joachim-breitner.de>:

Hi,

Am Sonntag, den 29.10.2017, 23:58 +0100 schrieb Sebastian Graf:

...

Hi,

just wanted to throw in the idea of parallelising the benchmark suite (hurts to even write that, but cachegrind) to speed up the build, if ever need be.

https://github.com/nomeata/gipeda/blob/master/ghc/run-speed.sh#L143

good idea indeed – I’ve been doing that since I started running cachegrind ;-)

Joachim

-- Joachim Breitner mail@joachim-breitner.de http://www.joachim-breitner.de/ _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs