#13564: Why does memory usage increase so much during CoreTidy? -------------------------------------+------------------------------------- Reporter: rwbarton | Owner: rwbarton Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by rwbarton): Happily the cause turns out to be simple. In `tidyUnfolding` we tidy the RHS of a stable `CoreUnfolding`. But `CoreUnfolding` also has fields `uf_is_value`, `uf_is_conlike`, etc., which are cached computations on the original RHS. Apparently it's common for these to never be used at any point (the iface will only record `uf_guidance`); so we should evaluate these in `tidyUnfolding`, or just throw away the ones that won't ever be used. I'm in the process of verifying this fix still has the expected result with HEAD. There might be further improvements to be made here, but indications are that this fix at least decreases memory usage during code generation to the point where it is no longer larger than the peak usage during simplification. In other words, max memory usage can't be reduced any more from this side. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13564#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13564: Why does memory usage increase so much during CoreTidy? -------------------------------------+------------------------------------- Reporter: rwbarton | Owner: rwbarton Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by rwbarton): * Attachment "ghc-stage2.2.pdf" added. heap profile with seqUnfolding in tidyUnfolding -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13564 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13564: Why does memory usage increase so much during CoreTidy? -------------------------------------+------------------------------------- Reporter: rwbarton | Owner: rwbarton Type: task | Status: new Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by rwbarton): * Attachment "ghc-stage2.3.pdf" added. heap profile with seqBinds in tidyTopBinds as well -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13564 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13564: Why does memory usage increase so much during CoreTidy? -------------------------------------+------------------------------------- Reporter: rwbarton | Owner: rwbarton Type: task | Status: patch Priority: normal | Milestone: Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D3516 Wiki Page: | -------------------------------------+------------------------------------- Comment (by Ben Gamari ): In [changeset:"5c602d2228d28530621cc6c94fbb736b13f474fb/ghc" 5c602d22/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="5c602d2228d28530621cc6c94fbb736b13f474fb" Avoid excessive space usage from unfoldings in CoreTidy Test Plan: validate Reviewers: austin, bgamari Reviewed By: bgamari Subscribers: thomie GHC Trac Issues: #13564 Differential Revision: https://phabricator.haskell.org/D3516 }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13564#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13564: Why does memory usage increase so much during CoreTidy? -------------------------------------+------------------------------------- Reporter: rwbarton | Owner: (none) Type: task | Status: new Priority: normal | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D3516 Wiki Page: | -------------------------------------+------------------------------------- Changes (by simonpj): * status: closed => new * owner: rwbarton => (none) * resolution: fixed => Comment:

I found that this spike went away with -v, so I added a seqBinds which also eliminates the spike

This is ok, but it's an un-satisfying Big Hammer. It forces us to make yet another pass over the entire program, when there is probably a more refined solution to hand. You might well be able to eliminate the other change in `tidyUnfolding` when you use the Big Hammer. And maybe we should eliminate it, since we are now seq'ing the unfolding twice. But it would be nicer to find the source of the spike in the first place. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13564#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13564: Why does memory usage increase so much during CoreTidy? -------------------------------------+------------------------------------- Reporter: rwbarton | Owner: (none) Type: task | Status: new Priority: normal | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D3516, Wiki Page: | Phab:D3524 -------------------------------------+------------------------------------- Comment (by Ben Gamari ): In [changeset:"b3da6a6c3546562d5c5e83b8af5d3fd04c07e0c1/ghc" b3da6a6c/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="b3da6a6c3546562d5c5e83b8af5d3fd04c07e0c1" CoreTidy: Don't seq unfoldings Previously we would force uf_is_value and friends to ensure that we didn't retain a reference to the pre-tidying template, resulting in a space leak. Instead, we now just reinitialize these fields (despite the fact that they should not have changed). This may result in a bit more computation, but most of the time we won't ever evaluate them anyways, so the damage shouldn't be so bad. See #13564. }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13564#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13564: Why does memory usage increase so much during CoreTidy? -------------------------------------+------------------------------------- Reporter: rwbarton | Owner: (none) Type: task | Status: new Priority: normal | Milestone: 8.4.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D3516, Wiki Page: | Phab:D3524 -------------------------------------+------------------------------------- Changes (by bgamari): * milestone: 8.2.2 => 8.4.1 Comment: Bumping this off to 8.4.1. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13564#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#13564: Why does memory usage increase so much during CoreTidy? -------------------------------------+------------------------------------- Reporter: rwbarton | Owner: (none) Type: task | Status: new Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #13586 | Differential Rev(s): Phab:D3516, Wiki Page: | Phab:D3524 -------------------------------------+------------------------------------- Changes (by MikolajKonarski): * related: => #13586 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13564#comment:14 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler