Re: RTS changes affect runtime when they shouldn’t

Hi, Am Samstag, den 23.09.2017, 20:45 +0200 schrieb Sven Panne:

2017-09-21 0:34 GMT+02:00 Sebastian Graf :

...

[...] The only real drawback I see is that instruction count might skew results, because AFAIK it doesn't properly take the architecture (pipeline, latencies, etc.) into account. It might be just OK for the average program, though.

It really depends on what you're trying to measure: The raw instruction count is basically useless if you want to have a number which has any connection to the real time taken by the program. The average number of cycles per CPU instruction varies by 2 orders of magnitude on modern architectures, see e.g. the Skylake section in ht tp://www.agner.org/optimize/instruction_tables.pdf (IMHO a must-read for anyone doing serious optimizations/measurements on the assembly level). And these numbers don't even include the effects of the caches, pipeline stalls, branch prediction, execution units/ports, etc. etc. which can easily add another 1 or 2 orders of magnitude.

So what can one do? It basically boils down to a choice:

* Use a stable number like the instruction count (the "Instructions Read" (Ir) events), which has no real connection to the speed of a program.

* Use a relatively volatile number like real time and/or cycles used, which is what your users will care about. If you put a non- trivial amount of work into your compiler, you can make these numbers a bit more stable (e.g. by making the code layout/alignment more stable), but you will still get quite different numbers if you switch to another CPU generation/manufacturer.

A bit tragic, but that's life in 2017... :-}

what I want to do is to reliably catch regressions. What are the odds that a change to the Haskell compiler (in particular to Core2Core transformations) will cause a significant increase in runtime without a significant increase in instruction count? (Honest question, not rhetoric). Greetings, Joachim -- Joachim Breitner mail@joachim-breitner.de http://www.joachim-breitner.de/