Hi, I did it for my thesis and I found it ok. I mean I always sent it off

to some machine and looked at the results later, so I did not really care whether it took 30mins or 2h.

I did the same for my thesis (the setup of which basically was a rip-off of Joachim's) and it was really quite bearable. I think it was even faster than doing NoFibRuns=30 without counting instructions. The only real drawback I see is that instruction count might skew results, because AFAIK it doesn't properly take the architecture (pipeline, latencies, etc.) into account. It might be just OK for the average program, though. On Wed, Sep 20, 2017 at 10:13 PM, Joachim Breitner

wrote:

Hi

Am Mittwoch, den 20.09.2017, 14:33 -0400 schrieb Ben Gamari:

...

Note that valgrind can also do cache modelling so I suspect it can give you a reasonably good picture of execution; certainly better than runtime. However, the trade-off is that (last I checked) it's incredibly slow. Don't you think I mean just a bit less peppy than usual. I mean soul-crushingly, mollasses-on-a-cold-December-morning slow.

If we think that we can bear the cost of running valigrind then I think it would be a great improvement. As you point out, the current run times are essentially useless.

I did it for my thesis and I found it ok. I mean I always sent it off to some machine and looked at the results later, so I did not really care whether it took 30mins or 2h.

I think I’ll try it in perf.haskell.org and see what happens.

Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de https://www.joachim-breitner.de/

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Hi, I have switched perf.haskell.org to run nofib with $ make -C nofib EXTRA_RUNTEST_OPTS=-cachegrind NoFibRuns=1 mode=slow -j8 Right now, a complete build with testsuite and nofib takes ~2½h, but that was before I added the "-j8" to the line above, so let’s see if that helps. Even with cachegrind, most nofib tests finish in under one minute, 7 take between one and 10 minutes, k-nucleotide needs 20 minutes and fannkuch-redux needs 27 minutes. All in all not too bad. I reset the perf.haskell.org database and started re-measuring commits from 055d73c6576bed2affaf96ef6a6b89aeb2cd2e9f on. It will take a while (a week maybe?) for it to catch up with master. I hope that this allows us to spot performance regressions with greater precision. Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de https://www.joachim-breitner.de/

2017-09-30 17:56 GMT+02:00 Joachim Breitner :

[...] I also wonder whether, when using cachegrind, the results from different machines are actually comparable. [...]

In general, they are not really comparable: cachegrind doesn't collect *actual* cache statistics, it emulates a simplified version of the caching machinery, trying to auto-detect parameters, see e.g.: http://valgrind.org/docs/manual/cg-manual.html#cg-manual.overview This doesn't mean that the numbers are useless, but they are only (good) general hints, and in rare extreme cases, they can be totally off from the real numbers. For the "real stuff", one has to use "perf", but then you can only compare numbers from the same CPU models.

Hi, Am Samstag, den 23.09.2017, 20:45 +0200 schrieb Sven Panne:

2017-09-21 0:34 GMT+02:00 Sebastian Graf :

...

[...] The only real drawback I see is that instruction count might skew results, because AFAIK it doesn't properly take the architecture (pipeline, latencies, etc.) into account. It might be just OK for the average program, though.

It really depends on what you're trying to measure: The raw instruction count is basically useless if you want to have a number which has any connection to the real time taken by the program. The average number of cycles per CPU instruction varies by 2 orders of magnitude on modern architectures, see e.g. the Skylake section in ht tp://www.agner.org/optimize/instruction_tables.pdf (IMHO a must-read for anyone doing serious optimizations/measurements on the assembly level). And these numbers don't even include the effects of the caches, pipeline stalls, branch prediction, execution units/ports, etc. etc. which can easily add another 1 or 2 orders of magnitude.

So what can one do? It basically boils down to a choice:

* Use a stable number like the instruction count (the "Instructions Read" (Ir) events), which has no real connection to the speed of a program.

* Use a relatively volatile number like real time and/or cycles used, which is what your users will care about. If you put a non- trivial amount of work into your compiler, you can make these numbers a bit more stable (e.g. by making the code layout/alignment more stable), but you will still get quite different numbers if you switch to another CPU generation/manufacturer.

A bit tragic, but that's life in 2017... :-}

what I want to do is to reliably catch regressions. What are the odds that a change to the Haskell compiler (in particular to Core2Core transformations) will cause a significant increase in runtime without a significant increase in instruction count? (Honest question, not rhetoric). Greetings, Joachim -- Joachim Breitner mail@joachim-breitner.de http://www.joachim-breitner.de/

On 2017-09-23 20:45, Sven Panne wrote:

2017-09-21 0:34 GMT+02:00 Sebastian Graf mailto:sgraf1337@gmail.com>:

[...] The only real drawback I see is that instruction count might skew results, because AFAIK it doesn't properly take the architecture (pipeline, latencies, etc.) into account. It might be just OK for the average program, though.

It really depends on what you're trying to measure: The raw instruction count is basically useless if you want to have a number which has any connection to the real time taken by the program. The average number of cycles per CPU instruction varies by 2 orders of magnitude on modern architectures, see e.g. the Skylake section in http://www.agner.org/optimize/instruction_tables.pdf (IMHO a must-read for anyone doing serious optimizations/measurements on the assembly level). And these numbers don't even include the effects of the caches, pipeline stalls, branch prediction, execution units/ports, etc. etc. which can easily add another 1 or 2 orders of magnitude.

So what can one do? It basically boils down to a choice:

   * Use a stable number like the instruction count (the "Instructions Read" (Ir) events), which has no real connection to the speed of a program.

   * Use a relatively volatile number like real time and/or cycles used, which is what your users will care about. If you put a non-trivial amount of work into your compiler, you can make these numbers a bit more stable (e.g. by making the code layout/alignment more stable), but you will still get quite different numbers if you switch to another CPU generation/manufacturer.

A bit tragic, but that's life in 2017... :-}

I may be missing something since I have only quickly skimmed the thread, but...: Why not track all of these things and correlate them with individual runs? The Linux 'perf' tool can retrieve a *lot* of interesting numbers, esp. around cache hit rates, branch predicition hit rates, etc. Regards,

2017-09-26 18:35 GMT+02:00 Ben Gamari :

While it's not a bad idea, I think it's easy to drown in information. Of course, it's also fairly easy to hide information that we don't care about, so perhaps this is worth doing regardless.

The point is: You don't know in advance which of the many performance characteristics "perf" spits out is relevant. If e.g. you see a regression in runtime although you really didn't expect one (tiny RTS change etc.), a quick look at the diffs of all perf values can often give a hint (e.g. branch prediction was screwed up by different code layout etc.). So I think it's best to collect all data, but make the user-relevant data (runtime, code size) more prominent than the technical/internal data (cache hit ratio, branch prediction hit ratio, etc.), which is for analysis only. Although the latter is a cause for the former, from a compiler user's perspective it's irrelevant. So there is no actual risk in drowning in data, because you primarily care only for a small subset of it.