[GHC] #9370: large blowup in memory usage and time when doing parallel build of xmlhtml package
#9370: large blowup in memory usage and time when doing parallel build of xmlhtml package -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.8.3 Keywords: | Operating System: Architecture: Unknown/Multiple | Unknown/Multiple Difficulty: Unknown | Type of failure: Blocked By: | None/Unknown Related Tickets: | Test Case: | Blocking: | Differential Revisions: -------------------------------------+------------------------------------- I've observed a blowup in ghc memory usage when invoked with parallel build flags. to reproduce {{{ cabal get xmlhtml-0.2.3.2 cabal install xmlhtml-0.2.3.2 --only-dependencies cd xmlhtml-0.2.3.2 }}} then {{{ cabal clean ; cabal configure --ghc-options="-j4" ; time cabal build -j4 -v3 }}} will take quite a while and use > 1gb of ram whereas {{{ cabal clean ; cabal configure --ghc-options="-j1 ; time cabal build -j1 -v3 }}} will use < 150mb of ram. Based upon the output of cabal build -j4 -v3, it looks like the parallel build is spending a lottt more time in the simplifier passes. I'm not sure what changes between the parallel and sequential builds to change this. I'll try to dig into this more in a few days, but recording this problem now before i forget. (though of course any insights would be appreciated) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: large blowup in memory usage and time when doing parallel build of xmlhtml package -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.8.3 Resolution: | Keywords: Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: | Blocked By: None/Unknown | Related Tickets: Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Changes (by ekmett): * cc: ekmett (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: large blowup in memory usage and time when doing parallel build of xmlhtml package -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: parmake Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: #910 #9221 Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Changes (by hvr): * keywords: => parmake * failure: None/Unknown => Compile-time performance bug * related: => #910 #9221 * milestone: => 7.10.1 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: large blowup in memory usage and time when doing parallel build of xmlhtml package -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: parmake Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: #910 #9221 Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by simonpj): Adding `-dshow-passes` will show you the size of the intermediates of each pass. Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: large blowup in memory usage and time when doing parallel build of xmlhtml package -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: parmake Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: #910 #9221 Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by rwbarton): Indeed something seems to be going wrong in the simplifier when invoking ghc with -j4. {{{ *** Checking old interface for xmlhtml-0.2.3.2:Text.XmlHtml.HTML.Meta: [ 1 of 10] Compiling Text.XmlHtml.HTML.Meta ( src/Text/XmlHtml/HTML/Meta.hs, dist/build/Text/XmlHtml/HTML/Meta.o ) *** Parser: *** Renamer/typechecker: *** Desugar: Result size of Desugar (after optimization) = {terms: 26,260, types: 20,021, coercions: 0} *** Simplifier: Result size of Simplifier iteration=1 = {terms: 1,446,658, types: 953,432, coercions: 314,352} ... }}} With `-ddump-inlinings` I see this repeated over and over, which is not present with `-j1`: {{{ Inlining done: Data.String.fromString Inlining done: Data.Text.$fIsStringText Inlining done: Data.Text.pack Inlining done: Data.Text.Internal.Fusion.unstream Inlining done: Data.Text.Internal.Fusion.Common.map Inlining done: Data.Text.Internal.Fusion.Common.streamList Inlining done: Data.Text.Internal.safe Inlining done: Data.Bits.$fBitsInt_$c.&. Inlining done: Data.Text.Internal.Fusion.Types.$WYield [ repeats ~4000 times ] }}} According to `-ddump-inlinings -dverbose-core2core`, GHC never even considered inlining `fromString` with `-j1`. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: large blowup in memory usage and time when doing parallel build of xmlhtml package -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: parmake Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: #910 #9221 Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by simonpj): That's bizarre. I can't think why the the simplifier would do ''anything'' different with `-j1` vs `-j4`! I'm hard pressed even to see where to start. Is it possible to determine when the log files start to differ? They should be identical except for the uniques (which you can suppress with `-dsuppress-uniques`). Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: OPTIONS_GHC (or its absence) leaks between modules compiled in parallel -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: parmake Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: #910 #9221 Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by rwbarton): Oh, the cause is hiding in plain sight, on the first line of `Text.XmlHtml.HTML.Meta`: {{{ {-# OPTIONS_GHC -O0 -fno-case-merge -fno-strictness -fno-cse #-} }}} If I add the same line to the start of the nine other modules in the package, then `Text.XmlHtml.HTML.Meta` compiles as quickly with `-j4` as with `-j1`. So somehow flags set by `OPTIONS_GHC` pragmas (or in this case, their absence) are leaking between different modules being built in parallel. These are all dynamic flags at least, right? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: OPTIONS_GHC (or its absence) leaks between modules compiled in parallel -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: normal | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: parmake Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: #910 #9221 Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by simonpj): Brilliant work! I have no idea about how the leakage is taking place... that will need someone who knows how the `-j` stuff works in GHC. But I ''am'' puzzled by the `OPTIONS_GHC` flags in `Text.XmlHtml.HTML.Meta`. That pretty much switches off all optimisation in GHC, and you should need a jolly good reason do to that. Indeed, the necessity of doing so signals either a bug in GHC, or else some bad rewrite rules that go into a loop. It would be good to know which. Is the `OPTIONS_GHC` stuff accompanied with a prominent comment to explain such a bizarre options line? If not, would someone care to investigate? (All this is separate from the flag leakage question, of course.) Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: OPTIONS_GHC (or its absence) leaks between modules compiled in parallel -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: highest | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: parmake Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: #910 #9221 Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Changes (by hvr): * priority: normal => highest Comment: This sounds kinda serious -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Changes (by rwbarton): * keywords: parmake => * priority: highest => high * related: #910 #9221 => Comment: OK I take it all back. The parallel build is not really responsible at all. You get the same blowup when building `Text.XmlHtml.HTML.Meta` with `-j1` if you just add an `import Text.XmlHtml.Common` so that `Text.XmlHtml.Common` is built before `Text.XmlHtml.HTML.Meta`. As far as I can tell, the reason is that when we read an interface file from an external package, we may or may not attach unfolding info to its Ids according to the dynamic flags that are in effect for the module we are building. But when we later build another module, we might reuse that external package information even though different dynamic flags are now in effect. I haven't tried to track down the code that loads interface files, but does this sound plausible? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by simonpj): Hmm. Is this with `ghc --make`? Then yes, that sounds plausible. * When compiling a module M with -O, and then reading an interface file to support that compilation, GHC makes the unfolding of Ids in that interface file. * With `ghc --make` when GHC goes on to compile a new module L, it doesn't re-read interface files it has already read (that is part of why `--make` is faster). So the already-read Ids still have those unfoldings in them. * As a result, if L is compiled with -O0, it will still see the unfoldings. I can see that is perplexing. If it's important, the solution would be to disable inlining for imported functions when -O0 is on. (Actually, more precisely, it's controlled by `-fignore-interface-pragmas`.) I don't think it would be terribly hard to do that, although it would be an extra test on every inlining for an imported function. I still wish someone could explain why it's so crucial for this module to be compiled with -O0. That must be a bug, either in the RULES for some package, or in GHC. Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by rwbarton): I would guess it's to avoid the very code size explosion (inlining 4000 copies of Text's `fromString`) that Carter encountered that originated this ticket. But that doesn't explain the `-fno-case-merge -fno-strictness -fno-cse` part. Carter has asked about this on the xmlhtml bug tracker. The same scenario could apply in reverse, right? In the actual xmlhtml package, `Text.XmlHtml.HTML.Meta` is the first module built and it is built with `-O0` so any interface files that are read while compiling that module will not have unfoldings attached. Then those modules will not have unfoldings during the compilation of any subsequent module either, even though those are built with `-O`. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by simonpj): Replying to [comment:11 rwbarton]:
I would guess it's to avoid the very code size explosion (inlining 4000 copies of Text's `fromString`)
The same scenario could apply in reverse, right? In the actual xmlhtml
I would guess so too. But WHY is 4000 copies of `fromString` getting inlined. I doubt GHC is doing that unaided. I bet it's an INLINE pragma or RULE in Text. And if so, it's a landmine waiting to kill new victims. package, `Text.XmlHtml.HTML.Meta` is the first module built and it is built with `-O0` so any interface files that are read while compiling that module will not have unfoldings attached. Then those modules will not have unfoldings during the compilation of any subsequent module either, even though those are built with `-O`. No, that part at least is not so. The interface file built for `Text.XmlHtml.HTML.Meta` will not have unfoldings in it, but that only affects functions actually defined in `Text.XmlHtml.HTML.Meta`. Later modules, built with `-O` will certainly get interface files with unfoldings in them. Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
The same scenario could apply in reverse, right? In the actual xmlhtml
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by rwbarton): Replying to [comment:12 simonpj]: package, `Text.XmlHtml.HTML.Meta` is the first module built and it is built with `-O0` so any interface files that are read while compiling that module will not have unfoldings attached. Then those modules will not have unfoldings during the compilation of any subsequent module either, even though those are built with `-O`.
No, that part at least is not so. The interface file built for
`Text.XmlHtml.HTML.Meta` will not have unfoldings in it, but that only affects functions actually defined in `Text.XmlHtml.HTML.Meta`. Later modules, built with `-O` will certainly get interface files with unfoldings in them. Sorry, I should have been more clear. I mean that when GHC builds `Text.XmlHtml.HTML.Meta`, causing it to read the interface file for `Data.String`, optimization level `-O0` is in effect, so GHC will not read the unfolding for `Data.String.fromString`. Then, when GHC builds the next module `Text.XmlHtml.Common`, even though it is now at optimization level `-O1`, it reuses its in-memory interface file information for `Data.String` and therefore has no unfolding available for `fromString`, so it cannot inline it. If we force GHC to build `Text.XmlHtml.Common` before `Text.XmlHtml.HTML.Meta` by adding an import of the former to the latter, then GHC does inline `fromString` in `Text.XmlHtml.Common`. You can observe this either with `-ddump-inlinings -dverbose-core2core`, or by examining the symbol table of the resulting object file: there is no reference to an undefined symbol `base_DataziString_fromString_info`. This doesn't just apply to the module `Data.String`, of course; we might be missing many other opportunities to inline when we build the rest of the modules in this package. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:14 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by rwbarton): (For anyone who is paying careful attention to #9400 and is wondering why we have ~4000 identical inlinings here while in that ticket we have only ~2000 copies of `unstream`, the reason is that here we are building with `-O0`, so we do not process the rule which rewrites `Data.Text.pack . unpackCString#` to `Data.Text.unpackCString#`! The text library does not expect its users to follow its INLINE directives without also following its RULES.) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
The text library does not expect its users to follow its INLINE
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by simonpj): Replying to [comment:15 rwbarton]: directives without also following its RULES.) And THAT is the GHC bug here: we are compiling a client with `-O0` but * we ''are'' obeying INLINE pragmas from `Text`, * we ''are not'' obeying RULES from `Text`. Right? Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:16 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by rwbarton): In the case that originated this ticket, yes. In general, the bug is that whether we obey INLINE pragmas in an imported module M depends not just on the optimization flags in effect for the module we are currently compiling, but also on the optimization flags that were in effect for the first module in this `ghc --make` invocation that imported M. (Or that caused GHC to load the interface file for M for any reason, if that ever happens other than because M was imported.) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 7.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: Operating System: | Architecture: Unknown/Multiple Unknown/Multiple | Difficulty: Unknown Type of failure: Compile- | Blocked By: time performance bug | Related Tickets: Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by simonpj): Right. I'm utterly swamped, but happy to advise. I think it would not be hard to fix this. Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:18 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 7.12.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Changes (by thoughtpolice): * milestone: 7.10.1 => 7.12.1 Comment: Looks like this one isn't changing in time for 7.10, so I'm moving it out. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 7.12.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by simonpj): Just to add: I think Reid's comment:14 is spot on. The solution, I believe, is this: * Always read in unfoldings, etc, including with `-O`. (Possible exception: one-shot mode.) * Check the `-fignore-interface-pragmas` flag when considering inlining a function. So an `Id` would always contain its inlining; but at the ''usage sites'' we'd decide whether or not to use it. An extra test at every inlining, but there are already a lot of tests at inlining sites! This would be a good, small, contained project for someone. Volunteers? If it's mission-critical for anyone, please yell. Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:20 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 7.12.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Revisions: -------------------------------------+------------------------------------- Changes (by rwbarton): * related: => #8635 Comment: This is pretty much the same issue as #8635. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomers Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * keywords: => newcomers * milestone: 8.0.1 => 8.2.1 Comment: Seems like a reasonable bite-sized project for a newcomer. Sadly, it won't be happening for 8.0.1, however. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:23 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomers Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by richardfung): * owner: => richardfung -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:24 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by mpickering): * keywords: newcomers => newcomer, Inlining Comment: Any progress on this ticket Richard? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:25 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by richardfung): Not really sadly.. I've been just trying to understand all the code. I started looking from where --make starts, trying to go down to where the ModInfo is actually generated. I see that mkIface_ generates the [IfaceDecl] that gets passed to addFingerprints which eventually returns the ModIface which we save and I believe reuse, but that's the farthest I've gotten; I still don't know where IfaceIdInfo is set to NoInfo. At least.. I think that's what's happening? Given the fact I haven't even started looking at the second part of this problem, checking when to inline, I'm beginning to wonder if I'm in over my head! Any advice would certainly be appreciated. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:26 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by mpickering): t is certainly hard work on the first ticket. I spent about an hour this morning trying to find out where the right part to change was and then read Ben's email earlier in this thread which made everything much clearer for me! The essential problem we are trying to solve is:
When a module is compiled with -O0, it does not read unfoldings from modules it imports. Because we reuse the unfolding information across modules (to avoid reloading interface files very often), this unfolding information is then not available in the rest of the compilation pipeline.
Simon's suggested fix is 1. Always read in unfoldings, etc, including with -O. (Possible exception: one-shot mode.) 2. Check the -fignore-interface-pragmas flag when considering inlining a function. So for part 1, we need to find the part which deals with *reading in* interface files and seeing where the unfolding information is lost. For part 2, we need to find the part of the optimiser which makes decisions about whether to inline a function. So now I'm re-reading Ben's comment and (as usual) his analysis is spot-on. If you trace the origin of the ignore_prags flag then you'll find it is set by checking whether the option `Opt_IgnoreInterfacePragmas` is set or not. You can find that this flag is automatically set when the opt-level is 0 in DynFlags, I'll leave you to find the exact location! It seems like Simon's suggested fix is removing this guard in loadDecls. It isn't clear to me whether the other information (other than the unfoldings) should still be read in or not. Simon will be able to comment. Now for part 2: I think you should start by looking at the callSiteInline function in coreSyn/CoreUnfold. That will at least get you started looking in the right place. Does that help? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:27 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): Matthew is right. * Part 1: the easiest thing is to set `Opt_IgnoreInterfacePragmas` to true ''only'' if you specify the flag `-fignore-interface-pragmas` explicitly, and ''not'' with `-O0`. Do that in `DynFlags`. That way an explicit `-fignore-interface-pragmas` flag will still work. * Part 2: with `O0` we want to switch off all inlining (except "compulsory" inlinings; see `InlineCompulsory` in CoreSyn). I think this can conveniently be done by setting `sm_inline = False` in the `SimplMode` in `SimplCore`. Set it in `base_mode`. So actually it looks fairly simple. Does that help? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:28 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by richardfung): Yes that helps tremendously! This change was a lot simpler than I expected.. I only wish I asked for help sooner. Thanks! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:29 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by richardfung): Sorry even though you basically did the ticket for me already I still have more questions.. I'm trying to think of a good test case to add for this but it seems like the tests are limited to comparing stdout and stderr and checking if it compiled or ran successfully, at least if I understand correctly. That being the case, I was thinking I could make a test that runs with -ddump-inlinings and compare the output. Would that work? Ideally I would like to run --make twice (with the scenario that someone else mentioned where we force the module with optimizations to run first and where it runs second) and compare the inlining output to make sure they're the same. This would be nice in case the -ddump-inlinings format ever changes but I'm not sure this kind of test is supported and I also don't know how likely it is that the format would change. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:30 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by ezyang): There are few cases where we have just directly checked in output to the test suite, grepping for `-ddump` I see `-ddump-simpl` (be sure to add `-dsuppress-uniques`), I also recently added a `-ddump-stg`. If you want to go your original approach, you'll need to write a `Makefile` rule for it. If you look at, for example, `testsuite/tests/driver/Makefile` you can see examples. Note that you'll have to edit `testsuite/driver/extra_files.py` to make sure all your test files get added -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:31 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by richardfung): Unless someone has a strong preference otherwise I'm inclined to go with the easier solution of comparing against a hard coded -ddump-inlinings output. I don't think I'd be able to manage the more complicated test without help and I'm not convinced it's worth the extra effort especially if we're already doing things the simpler way elsewhere. Where should these tests go though? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:32 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): Tests should got in `testsuite/tests/simplCore` -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:33 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by richardfung): Am I correct in thinking the right way to test this is to write a function which would only be inlined if the INLINE pragma is there? Otherwise, the function would be inlined even in source files that import it with OPTIONS_GHC -O0 right? I was completely unfamiliar with inlining before this ticket so please bear with me. It seems like imported modules are still inlined even if OPTIONS_GHC -O0 is set if you run ghc --make -O. The documentation also says that -O enables cross module inlining so I'm guessing even if the source file has OPTIONS_GHC -O0, inlinings from other modules are still supposed to happen? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:34 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Sorry I missed your questions, richardfung. Regardless, thanks for your work! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:35 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by richardfung): No problem at all, I managed to figure it out on my own! Thank you for all the help so far. I suspect I'm overthinking this or am missing something, but won't we still have this issue if we have multiple modules but one explicitly specifies -fignore-interface-pragmas? My understanding is that the issue can manifest itself in two ways: either we load an optimized module first and unoptimized second, thus using inlinings even with -O0, or unoptimized first and optimized second, in which case we should inline with -O but we can't because we did not read in the pragmas. Couldn't the second case still happen with the suggested fix if the unoptimized module also specified -fignore-interface-pragmas explicitly? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:36 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj):
I suspect I'm overthinking this or am missing something, but won't we still have this issue if we have multiple modules but one explicitly specifies -fignore-interface-pragmas?
This has been a long thread and I'm having trouble understanding your question. E.g. what is "this issue" in the above? Would you like to ask the question again, from scratch as it were, illustrated by a concrete example? And presumably when you say "won't we ''still'' have" you mean "won't we still have it when we have done X". But what precisely is X? Perhaps you could lay that out too. Then your question wouldn't depend on having paged in 36 previous comments :-). Thanks Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:37 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by richardfung): Ah yes sorry I shall try to keep my comments more self contained. The ticket addresses the issue that the unfolding info seen depends on previously compiled modules with --make. As I understand it, this can either cause ghc to inline when it shouldn't (in -O0) or to be unable to inline when it should. For example, if we have modules A and B, both which import module C, and we assume A is built first, then whether B sees the interface pragmas of C is dependent on whether they were read in when compiling A. If A is built with -O0, then even if B is built with -O it can not inline things from C because it doesn't have the unfolding info. Alternatively, if A is built with -O and B is built with -O0, B will still see the unfolding info and use it. The suggested fix is to disable -fignore-interface-pragmas being implied by -O0 and to make sm_inline = False when -O0. I do believe this solves both of the scenarios described above, but I'm concerned that if A is built with -fignore-interface-pragmas, and B is built with -O, B will still not have the unfolding info from C when compiling. This would be identical to what's currently happening when A is built with -O0 and B with -O. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:38 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): You're right. Maybe we should move `-fignore-interface-pragmas` to `-dignore-interface-pragmas`; that is, make it a debug flag not intended for generic users but rather for people who know what they are doing. Or else remove it altogether. I don't mind which; I have never used it myself. Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:39 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by richardfung): There has been some discussion on the code review: https://phabricator.haskell.org/D2485 To summarize, turning off all inlining with -O0 and always reading in interface pragmas causes a performance regression. Simon Marlow has suggested two other solutions: "Lazily load the pragma info, so that it doesn't cost anything if we don't use it. The simplifier should use -fignore-interface-pragmas to decide whether to use the pragma info from external Ids or not. Predict whether we'll need the pragma info by determining whether *any* module in the current set will be compiled with optimisation turned on. This info is available after we've done the downsweep in the compilation manager. This approach doesn't really work in general because a user of the GHC API might load more modules later with -O and we can't predict that, but it fixes the common case." I would rather do things the right way and thus try my hand at the first solution. Is this something that is still suitable for a newcomer? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:40 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: richardfung Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari):
I would rather do things the right way and thus try my hand at the first solution. Is this something that is still suitable for a newcomer?
I'm happy to hear that you are willing to give it another go. Regarding Simon's suggestion, I suspect you could pull it off, although it will naturally require some learning. You'll likely want to start reading `compiler/iface/LoadIface.hs`. In particular pay attention to the `ignore_prags` argument to `loadDecl`. This is how we currently tell the typechecker not to bother typechecking unfoldings. You will likely want to remove this; instead you want to fork off typechecking as an interleaved computation with `TcRnMonad.forkM`. Then just make sure that there are no unconditional strictness demands on the pragma field and add a condition on `-fignore-interface-pragmas` to ignore the presence of the pragmas when necessary. As always, let me know if I can be of help, especially if any of the above seems incorrect; it is largely the result of a cursory glance over the relevant code and I could have easily missed something. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:41 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by richardfung): * owner: richardfung => -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:42 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Thanks Richard! I hope we see you again when you have time for some hacking. I had a look at this and the result is Phab:D2609. There is a bit more to it that comment:41 suggests. In addition to what I describe there I also needed to track down a few places outside of the simplifier where unfoldings were being force. The first place I encountered was `OccurAnalyse`, which wants to look a free variables of unfoldings. There are two ways I can envision fixing this, * Have occurrence analysis pretend there are no unfoldings if `-fignore- interface-pragmas` is passed. Note that this is a bit of a lie: we would be ignoring all unfoldings, not just those from interface files (since we can't tell the difference between the two under the current scheme). * Skip occurrence analysis altogether. If we are going to disable inlining entirely then (I think) it serves little purpose. Simon, perhaps you could weigh in here? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:43 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): Actually we really don't want to disable ALL unfoldings, just unfoldings for imported things. Locally-bound things (esp in local lets) should be fine. I suppose that means that every use of `idUnfolding` has to be guarded with a test of `DynFlags`. The "is it imported?" test can be `isLocalVar`. I took a look at uses of `idUnfolding`. * Some, like `idUnfoldingFVs` can be dealt with by testing for `isLocalVar`. Global vars will have no fvs, so no point in looking at the unfolding. This might even improve perf! * Some, like the use of `idUnfolding` in `CoreLint` are guaranteed to be on a `LocalId`, so that can be an assertion. * The`isCompulsoryUnfolding` in `CoreSubst` is OK; we do want to inline conpulsory unfoldings regardless. Etc. It's tiresome to look at all these call sites, but I see no alternative. We have an Id that does ''have'' an unfolding, but we don't want to look at it.... -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:44 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by nh2): * cc: nh2 (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:45 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.2.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * owner: => bgamari Comment: I have an in-flight patch for this. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:46 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * milestone: 8.2.1 => 8.4.1 Comment: Sadly I didn't have a chance to finish this patch for 8.2. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:47 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by duog): bgamari: What is your approach for fixing this? Do you think it could be adapted to address the hs-boot visibility problems in trac:14092? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:48 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.6.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * milestone: 8.4.1 => 8.6.1 Comment: This won't happen for 8.4.
bgamari: What is your approach for fixing this? Do you think it could be adapted to address the hs-boot visibility problems in trac:14092?
Essentially it is to find all of the places where we use unfoldings and make them conditional on `DynFlags`. Then we can eliminate the conditional reading of unfoldings from interface files. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:49 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#9370: unfolding info as seen when building a module depends on flags in a previously-compiled module -------------------------------------+------------------------------------- Reporter: carter | Owner: RolandSenn Type: bug | Status: new Priority: high | Milestone: 8.10.1 Component: Compiler | Version: 7.8.3 Resolution: | Keywords: newcomer, | Inlining Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #8635 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by RolandSenn): * owner: bgamari => RolandSenn Comment: [https://ghc.haskell.org/trac/ghc/ticket/13002#comment:18 | See ticket #13002 comment 18] -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9370#comment:52 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC