Joachim Breitner schrieb:

This runs on a dedicated physical machine, and still the run-time
numbers were varying too widely and gave us many false warnings (and
probably reported many false improvements which we of course were happy
to believe). I have since switched to measuring only dynamic
instruction counts with valgrind. This means that we cannot detect
improvement or regressions due to certain low-level stuff, but we gain
the ability to reliably measure *something* that we expect to change
when we improve (or accidentally worsen) the high-level
transformations.

While this matches my experience with the default settings, I had good results by tuning the number of measurements nofib does.
With a high number of NoFibRuns (30+) , disabling frequency scaling, stopping background tasks and walking away from the computer
till it was done I got noise down to differences of about +/-0.2% for subsequent runs.

This doesn't eliminate alignment bias and the like but at least it gives fairly reproducible results.

Sven Panne schrieb:

4% is far from being "big", look e.g. at https://dendibakh.github.io/blog/2018/01/18/Code_alignment_issues where changing just the alignment of the code lead to a 10% difference. :-/ The code itself or its layout wasn't changed at all. The "Producing Wrong Data Without Doing Anything Obviously Wrong!" paper gives more funny examples.

I'm not saying that code layout has no impact, quite the opposite. The main point is: Do we really have a benchmarking machinery in place which can tell you if you've improved the real run time or made it worse? I doubt that, at least at the scale of a few percent. To reach just that simple yes/no conclusion, you would need quite a heavy machinery involving randomized linking order, varying environments (in the sense of "number and contents of environment variables"), various CPU models etc. If you do not do that, modern HW will leave you with a lot of "WTF?!" moments and wrong conclusions.

You raise good points. While the example in the blog seems a bit constructed with the whole loop fitting in a cache line the principle is a real concern though.
I've hit alignment issues and WTF moments plenty of times in the past when looking at micro benchmarks.

However on the scale of nofib so far I haven't really seen this happen. It's good to be aware of the chance for a whole suite to give
wrong results though.
I wonder if this effect is limited by GHC's tendency to use 8 byte alignment for all code (at least with tables next to code)?
If we only consider 16byte (DSB Buffer) and 32 Byte (Cache Lines) relevant this reduces the possibilities by a lot after all.

In the particular example I've hit however it's pretty obvious that alignment is not the issue. (And I still verified that).
In the end how big the impact of a better layout would be in general is hard to quantify. Hence the question if anyone has
pointers to good literature which looks into this.

Cheers
Andreas