performance regressions

Hi devs, Phab has shown up some performance regressions in my recent commits. See https://phabricator.haskell.org/harbormaster/build/2607/. The failures except for haddock.base are new, and evidently my fault. They didn't show up on Travis. Will look into it shortly, but I doubt over the weekend. I don't think this should hold up the 7.10 fork, however. I have a suspicion as to what caused the problem -- the reimplementation of TcInteract.matchFam. That reimplementation was solely to reduce code repetition between TcInteract.matchFam and FamInstEnv.reduceTyFamApp_maybe. It can safely be rolled back -- just use the implementation of matchFam that existed before my commits. I'll look into this Monday at the latest, but if a remedy is needed sooner, try the technique above. Why would these failures show up on Phab but not on Travis? Sorry about the bother! Thanks, Richard

Hi, Am Freitag, den 12.12.2014, 21:51 -0500 schrieb Richard Eisenberg:
Phab has shown up some performance regressions in my recent commits. See https://phabricator.haskell.org/harbormaster/build/2607/. The failures except for haddock.base are new, and evidently my fault. They didn't show up on Travis. Will look into it shortly, but I doubt over the weekend.
ghcspeed also observes this: http://ghcspeed-nomeata.rhcloud.com/changes/?rev=7256213843b80d75a86f033be77516a62d56044a&exe=2&env=johan%27s%20buildbot Especially the T9872 benchmarks have a huge increase in allocations. But you seem to be aware of this, so that’s fine. Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org

I think I've fixed this. I've pushed the fix to wip/rae, and waiting for validation results before pushing to master.
My hunch below was right -- it was the change to matchFam, which essentially evaluated type-level functions more strictly. I've now made it lazier again. I'd like to better understand the tradeoff here, and to see if there's a principled sweet spot. But that will happen in a few days.
Expect a push to master soon.
Again, sorry for the bother.
Richard
On Dec 13, 2014, at 8:32 AM, Joachim Breitner
Hi,
Am Freitag, den 12.12.2014, 21:51 -0500 schrieb Richard Eisenberg:
Phab has shown up some performance regressions in my recent commits. See https://phabricator.haskell.org/harbormaster/build/2607/. The failures except for haddock.base are new, and evidently my fault. They didn't show up on Travis. Will look into it shortly, but I doubt over the weekend.
ghcspeed also observes this: http://ghcspeed-nomeata.rhcloud.com/changes/?rev=7256213843b80d75a86f033be77516a62d56044a&exe=2&env=johan%27s%20buildbot
Especially the T9872 benchmarks have a huge increase in allocations. But you seem to be aware of this, so that’s fine.
Greetings, Joachim
-- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org
_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Fixed, hopefully!
On Dec 13, 2014, at 10:03 AM, Richard Eisenberg
I think I've fixed this. I've pushed the fix to wip/rae, and waiting for validation results before pushing to master.
My hunch below was right -- it was the change to matchFam, which essentially evaluated type-level functions more strictly. I've now made it lazier again. I'd like to better understand the tradeoff here, and to see if there's a principled sweet spot. But that will happen in a few days.
Expect a push to master soon.
Again, sorry for the bother.
Richard
On Dec 13, 2014, at 8:32 AM, Joachim Breitner
wrote: Hi,
Am Freitag, den 12.12.2014, 21:51 -0500 schrieb Richard Eisenberg:
Phab has shown up some performance regressions in my recent commits. See https://phabricator.haskell.org/harbormaster/build/2607/. The failures except for haddock.base are new, and evidently my fault. They didn't show up on Travis. Will look into it shortly, but I doubt over the weekend.
ghcspeed also observes this: http://ghcspeed-nomeata.rhcloud.com/changes/?rev=7256213843b80d75a86f033be77516a62d56044a&exe=2&env=johan%27s%20buildbot
Especially the T9872 benchmarks have a huge increase in allocations. But you seem to be aware of this, so that’s fine.
Greetings, Joachim
-- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org
_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Hello Richard, Can you please push the fix asap to master? This performance failures are causing distracting false alarms (in terms of validation failures) on each pushed commit as well as submitted code-revisions. Thanks, HVR On 2014-12-13 at 16:55:40 +0100, Richard Eisenberg wrote:
Fixed, hopefully!
On Dec 13, 2014, at 10:03 AM, Richard Eisenberg
wrote: I think I've fixed this. I've pushed the fix to wip/rae, and waiting for validation results before pushing to master.
My hunch below was right -- it was the change to matchFam, which essentially evaluated type-level functions more strictly. I've now made it lazier again. I'd like to better understand the tradeoff here, and to see if there's a principled sweet spot. But that will happen in a few days.
Expect a push to master soon.
Again, sorry for the bother.

Hi, Am Samstag, den 13.12.2014, 10:55 -0500 schrieb Richard Eisenberg:
Fixed, hopefully!
Mitigated, but still a regression: http://ghcspeed-nomeata.rhcloud.com/timeline/?exe=2&base=2%2B68&ben=tests%2Falloc%2FT9872a&env=1&revs=50&equid=on# Is that now a level that we’ll have to live with, or is it still unexpectedly high? Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org

I pushed my supposed fix yesterday morning, as I emailed out the "Fixed, hopefully" note.
Of course, I now see that it wasn't a full fix.
This is all most assuredly my fault. However, I also feel that elements of the infrastructure failed me somewhat, making this error easier for me to commit:
- Travis has not picked up on these errors.
- Harbormaster has seemed unreliable, finding spurious compile-time errors sometimes, and sometimes just failing to test code that it sees. For example, when I pushed Diff 1901 to D546, Harbormaster never even attempted. Also, when I pushed my "fix", commit 3ec9391711, Harbormaster also skipped, as viewable here: https://phabricator.haskell.org/diffusion/GHC/
So, after pushing yesterday morning, I didn't get any email from Harbormaster saying that it failed, so I thought my fix was indeed a fix. Because my local build was a devel2 build (and that I had only about 20 minutes to work), I was unable to test locally -- all I could tell is that my fix lowered the numbers (as verified by ghcspeed).
Having a weekend full of plans, there wasn't really any opportunity for me to do the work necessary to figure out what's going on. It will be first on my docket tomorrow.
I suppose one lesson here is that I shouldn't push anything at all non-trivial on a Friday afternoon. But I also hope we can either improve the infrastructure (of course, it's *much* better than it was months ago!) or have realistic expectations of what we can expect from the infrastructure (e.g., trust Harbormaster/Travis when seeking feedback, but always validate locally before actually pushing to master).
More tomorrow,
Richard
On Dec 14, 2014, at 3:47 PM, Joachim Breitner
Hi,
Am Samstag, den 13.12.2014, 10:55 -0500 schrieb Richard Eisenberg:
Fixed, hopefully!
Mitigated, but still a regression:
Is that now a level that we’ll have to live with, or is it still unexpectedly high?
Greetings, Joachim
-- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org

Hi, Am Sonntag, den 14.12.2014, 22:37 -0500 schrieb Richard Eisenberg:
Of course, I now see that it wasn't a full fix.
This is all most assuredly my fault.
I wouldn’t call it a fault. Where wood is chopped, splinters must fall. (Hopefully correct translation of a German idiom.) We don’t _have_ to catch everything before it hits master, it is already pretty good if we manage to catch and fix regressions later. I guess we could use the known_broken feature of the test suite more quickly, to signal that an issue is being worked on and to avoid others from stumbling over test suite failures.
- Travis has not picked up on these errors.
unfortunately, travis is slighly less useful since a few weeks due to T5681 failing (possibly due to the use of LLVM-3.4), but I’m still waiting for an reply on that issue. But it wouldn’t have helped you: Travis generally skips all performance tests. These used to be far less reliable when I set up travis (this got better due to the removal of some of the max_bytes_allocated tests, and also due to harbormaster and ghcspeed monitoring), and also because we still hardly make the 50 minute deadline on Travis. So do not rely on Travis for performance measurements. (This is documented on https://ghc.haskell.org/trac/ghc/wiki/Travis, but it needs to become common knowledge.) Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • http://www.joachim-breitner.de/ Jabber: nomeata@joachim-breitner.de • GPG-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org
participants (3)
-
Herbert Valerio Riedel
-
Joachim Breitner
-
Richard Eisenberg