#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): I'm actually not sure that the effect on the whole is all that modest, unfortunately. [[https://perf.haskell.org/ghc/#revision/8d5cf8bf584fd4849917c29d82dcf46ee75dd035|Gipeda]] shows that the build time for GHC itself went up by 13%. While some increase is to be expected, this is quite significant. Are there any obvious knobs that could be adjusted to reduce the amount of work that the join point analysis/simplifier do under `-O1`? The full analysis is great for `-O2`, but I do fear that users will be up in arms if the GHC regression is representative of user code. Also, strangely enough, while the join points work eliminated all allocations from `fannkuch-redux`, the actual runtime of this test might have actually gone up by 8%. I say "might have" since runtime measurements are notoriously unstable, but it does look likely that this is a real regression: the test's runtime is quite long (4 seconds) and 8% is quite a large change. Even if this measurement were off by a factor of two we would want to understand it. Can you shed any light on this, Luke? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj):

Gipeda shows that the build time for GHC itself went up by 13%

Well that is new and very unwelcome information. Some of the `compiler/perf` tests went down quite a bit, a couple increased. So I was under the impression that compile time impact was slight, and some runtime improvements were jolly big. But a 13% increase across the entire build is massive. Can we get any insight into what is increasing? It's hard to konw where to start with a single number across the entire build. A good place to look might be binary sizes: if we generate more code then that'll take longer to compile. Bother. Still, there is no good reason that any of this join point stuff should cost much more. Yes, the occurrence analyser is doing a bit more work, but not much. If occ-anal cost 13% more I'd be surprised. But it'd have to cost 100% more to make the entire build take 13% more. Something is fishy. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonmar): What about looking at compiler allocations for nofib, to try to get a smaller example? I think this should be an 8.2 release blocker. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by dobenour): Is the bootstrap time regression due to a bug I introduced in Phab:D2879 that was fixed by rwbarton in Phab:D3070? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): Reid, Ben, David: might one of you be able to dig into where the 13% build time increase is coming from? The compiler/perf tests suggest that it's not an across-the-board increase in compile time, so perhaps some things are big outliers? Without more insight we are stuck on this, and we can hardly release with such a big step change in build time -- at least not without understanding it. Thanks -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

I'm actually not sure that the effect on compiler performance is all

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by lukemaurer): Replying to [comment:2 bgamari]: that modest, unfortunately. [[https://perf.haskell.org/ghc/#revision/8d5cf8bf584fd4849917c29d82dcf46ee75dd035|Gipeda]] shows that the build time for GHC itself went up by 13%. While some increase is to be expected, this is quite significant. Are there any obvious knobs that could be adjusted to reduce the amount of work that the join point analysis/simplifier do under `-O1`? The full analysis is great for `-O2`, but I do fear that users will be up in arms if the GHC regression is representative of user code. There isn't anything obvious—the occurrence analyser looks for join points as it goes, and that process should only be a small constant overhead. Nothing in the simplifier (or elsewhere that I can think of) has to work particularly hard to handle join points (or it shouldn't, anyway!).

Also, strangely enough, while the join points work eliminated all allocations from `fannkuch-redux`, the actual runtime of this test might have actually gone up by 8%. I say "might have" since runtime measurements are notoriously unstable, but it does look likely that this is a real regression: the test's runtime is quite long (4 seconds) and 8% is quite a large change. Even if this measurement were off by a factor of two we would want to understand it. Can you shed any light on this, Luke?

I haven't seen that jump before, and it's not obvious why it would happen. A quick diff of the dumps shows that the two versions end up with wildly different code, so it may be hard to pin any particular culprit. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by rwbarton): An initial report. I collected timing information from validation builds of the "Join points" commit and its parent commit, each built twice using ghc-8.0.2 and then a perf build of the same commit as the bootstrapping compiler (for reasons to become clear below). I have only looked at the timing results for a few modules so far. Here are the times for everyone's favorite module, `DynFlags.hs`. {{{ stage1 build: bootstrapped by mutator CPU seconds ghc-8.0.2 63s ghc-HEAD-before-join 63s ghc-HEAD-after-join 67s stage2 build: ghc-HEAD-before-join 150-154s ghc-HEAD-after-join 209-213s }}} Observations: * Performance of the perf build (used as the bootstrapping compiler, for the stage1 build of `DynFlags`) may have regressed by a few percent with join points. For now this is looking at the time for just one module, so I wouldn't take that number very seriously. * The time it took the stage1 compiler to perform the stage2 build of `DynFlags` increased by around 40% with the join point change. But, these numbers are both MUCH larger than the time it took to do the stage1 build of `DynFlags`. There are a few possible causes: - The stage1 compiler is built with only `-O`, unlike the perf build that produced the bootstrapping compiler, which built with `-O2`. - The stage1 compiler is built with `-DDEBUG`, so it contains extra sanity checks. - The stage2 compiler is built with `-dcore-lint`, so additional extra checks are being done during the stage2 build. - Something else about the build could be different between the two stages, like the values of CPP macros. I think this is unlikely to be the cause in this case. One of these points seems to have gotten much worse with join points. The good news is that, according to preliminary results, the performance of the perf (i.e., release) build of the compiler seems to be minimally affected. And if the performance regression is in `-dcore-lint`, for example, it may be easy to track down and eliminate. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:14 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): When GHC takes longer to compile the same file, it's sometime because it's generated bigger code. What does `-dshow-passes` show `ghc-before` and `ghc-after` compiling the same `DynFlags.hs`? `DynFlags` is a jolly big file. Are there smaller modules that show a big compile-time difference. `nofib-analyse` shows some per-module info I think. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:16 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by lukemaurer): Thanks—glad to hear it was (so far) a simple fix … -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:18 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by Simon Peyton Jones ): In [changeset:"6bab649bde653f13c15eba30d5007bef4a9a9d3a/ghc" 6bab649b/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="6bab649bde653f13c15eba30d5007bef4a9a9d3a" Improve checking of joins in Core Lint This patch addresses the rather expensive treatment of join points, identified in Trac #13220 comment:17 Before we were tracking the "bad joins". Now we track the good ones. That is easier to think about, and much more efficient; see CoreLint Note [Join points]. On the way I did some other modest refactoring, among other things removing a duplicated call of lintIdBndr for let-bindings. On teh }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:20 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): Status on this: Reid/David are going to look into why so much time is going into Lint. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:22 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): [[https://perf.haskell.org/ghc/#revision/6bab649bde653f13c15eba30d5007bef4a9a9d3a|Gipeda]] suggests that the patch didn't have much of an effect. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:24 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by dfeuer): * cc: dfeuer (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:26 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by rwbarton): Okay, found the REAL culprit. It's embarrassingly simple, and not at all what I had been looking for. Running `ghc` with `-dcore-lint -v` and grepping for `Core Linted result` revealed the issue, this new line in `simplifyPgmIO`: {{{#!diff Err.dumpIfSet_dyn dflags Opt_D_dump_occur_anal "Occurrence analysis" (pprCoreBindings tagged_binds); + lintPassResult hsc_env CoreOccurAnal tagged_binds; -- Get any new rules, and extend the rule base -- See Note [Overall plumbing for rules] in Rules.hs }}} We now run Core Lint twice per simplifier iteration, which in my sample program amounted to running Core Lint 50% more times overall. That's rather expensive. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:28 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): BTW, do you withdraw comment:25? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:30 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by rwbarton): Replying to [comment:30 simonpj]:

BTW, do you withdraw comment:25?

Well, the addition of that `ensureEqTys` did contribute a couple percent to the total compilation time (with `-dcore-lint`) in my test. If it's useful we can keep it; I don't understand what kind of substitution `le_subst` Core Lint is carrying around, so it's unclear to me whether this is actually testing something about the program being compiled or testing the consistency of Core Lint itself. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:32 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:3391 Wiki Page: | -------------------------------------+------------------------------------- Changes (by rwbarton): * differential: => Phab:3391 Comment: Phab:3391 is for removing the added Core Lint pass after occurrence analysis. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:34 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: lukemaurer Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:3391 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ben Gamari ): In [changeset:"71916e1c018dded2e68d6769a2dbb8777da12664/ghc" 71916e1c/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="71916e1c018dded2e68d6769a2dbb8777da12664" Remove Core Lint pass on occurrence analysis output (#13220) It was expensive, as the simplifier runs for many iterations, and probably not very useful. Test Plan: harbormaster Reviewers: austin, bgamari, dfeuer Reviewed By: dfeuer Subscribers: dfeuer, thomie Differential Revision: https://phabricator.haskell.org/D3391 }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:36 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:3391 Wiki Page: | -------------------------------------+------------------------------------- Changes (by simonpj): * owner: lukemaurer => (none) * status: closed => new * resolution: fixed => Comment: comment:31 not yet attended to, so re-opening. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:38 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: rwbarton Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:3391 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * owner: (none) => rwbarton Comment: Reid, do you suppose you could have a look at the remaining regressions when you get a chance? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:40 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13220: Performance regressions in testsuite from join points -------------------------------------+------------------------------------- Reporter: lukemaurer | Owner: rwbarton Type: bug | Status: new Priority: normal | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: JoinPoints Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:3391 Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): Fair enough. Shall we close this, then? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13220#comment:42 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler