Hi, Am Freitag, den 09.12.2016, 13:54 +0800 schrieb Moritz Angermann:
I am not sure what you are saying. Are you proposing the maintain a benchmark set outside GHC, or did you get the impression that I am proposing it?
Yes, that’s what *I* am proposing for the reasons I mentioned; one I did not yet mention is time. Running nofib takes time, adding more time consuming performance tests would reduce their likelihood of being run in my experience. As I see this as being almost completely scriptable, this could live outside of ghc i think.
I don’t think the running time of nofib is a constraint at the moment, and I expect most who run nofib to happily let it run for a few minutes more in order to get more meaningful results.
But maybe it is ok if it part of nofib, and hence of GHC, so that every breaking change in GHC can immediately be accounted for in the benchmark code.
A nice side effect of this might be that GHC developers can get a better idea of how much code their change breaks.
I’m not much a fan of this, but that’s just my opinion :-)
What is the alternative? Keep updating the libraries? But libraries change APIs. Then you need to keep updating the program itself? That seems to be too many moving parts for a benchmark suite.
Something I’ve recently had some success with was dumping measurements into influxdb[1] (or a similar data point collections service) and hook that up to grafana[2] for visualization.
Nice! Although these seem to be tailored for data-over-time, not data-over-commit. This mismatch in the data model was part of the motivation for me to create gipeda, which powers https://perf.haskell.org/ghc/
Assuming we confine this to a particular branch, or discriminate by branch, commits would be measured in sequence anyway, and the timestamp could be the time of the reporting of the measurement, and the respective ghc commit hash end up being an annotation. While this is not very pretty (and I would hope that grafana has some other ability to enrich the hover-tooltips) it could present a flexible solution without requiring additional engineering effort.
However, if gipeda is sufficient, please ignore my comment :)
Oh, we could certainly do better here! (But it serves my purposes for now, so I’ll stick to it until someone sets up something better, in which case I happily dump my code.) Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • https://www.joachim-breitner.de/ XMPP: nomeata@joachim-breitner.de • OpenPGP-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org