[GHC] #13535: vector test suite uses excessive memory on GHC 8.2
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: Compile-time Unknown/Multiple | performance bug Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- First noticed [https://github.com/haskell/vector/pull/161#issuecomment-292031845 here]. I haven't managed to boil this down to a test case with no dependencies yet, so for the time being, this requires `vector`. To reproduce, follow these steps: {{{ $ git clone https://github.com/erikd/vector $ cd vector/ $ cabal install --only-dependencies --enable-tests -w /opt/ghc/8.2.1/bin/ghc $ cabal configure --enable-tests -w /opt/ghc/8.2.1/bin/ghc $ cabal test }}} When building `vector-tests-O2`, GHC will stall when compiling the `Tests.Vector` module. On machines with modest memory allowances (e.g., [https://travis-ci.org/haskell/vector/jobs/218749281#L1270 the machines used on Travis CI]), GHC will be killed with an out-of-memory error after trying to compile `Tests.Vector` for a while. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by RyanGlScott): In particular, limiting GHC's memory to 4 GB is sufficient: {{{ $ bash -c 'ulimit -v 4000000; cabal test' ... Preprocessing test suite 'vector-tests-O2' for vector-0.12.0.1... [6 of 7] Compiling Tests.Vector ( tests/Tests/Vector.hs, dist/build /vector-tests-O2/vector-tests-O2-tmp/Tests/Vector.o ) tests/Tests/Vector.hs:9:1: warning: [-Wunused-imports] The import of ‘Data.Foldable’ is redundant except perhaps to import instances from ‘Data.Foldable’ To import instances alone, use: import Data.Foldable() | 9 | import Data.Foldable (Foldable(foldMap)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/Tests/Vector.hs:25:1: warning: [-Wunused-imports] The import of ‘Data.Monoid’ is redundant except perhaps to import instances from ‘Data.Monoid’ To import instances alone, use: import Data.Monoid() | 25 | import Data.Monoid | ^^^^^^^^^^^^^^^^^^ tests/Tests/Vector.hs:29:1: warning: [-Wunused-imports] The import of ‘Data.Functor.Identity’ is redundant except perhaps to import instances from ‘Data.Functor.Identity’ To import instances alone, use: import Data.Functor.Identity() | 29 | import Data.Functor.Identity | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/Tests/Vector.hs:438:5: warning: [-Wname-shadowing] This binding for ‘limitUnfolds’ shadows the existing binding imported from ‘Utilities’ at tests/Tests/Vector.hs:5:1-24 | 438 | limitUnfolds f (theirs, ours) | ^^^^^^^^^^^^ ghc: out of memory }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by rwbarton): Looks like a regular "Core program gets very large" issue. The Core program got very large in 8.0.1 too (I didn't try earlier versions) but now it gets even more very large. I tried with the versions of GHC before and after the Join points commit since I have them lying around. Already before the Join points commit the Core program grows to be very large quickly (actually, even more so than with HEAD). -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by George): * cc: george (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by asr): * cc: asr (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * cc: bgamari (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * related: => #10800 Comment: Note that #10800 is another ticket reporting a compile-time regression in `vector`'s testsuite between GHC 7.8 and 7.10. While it was apparently resolved in GHC 8.0, we never were able to come to a conclusion on what fixed it. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2
-------------------------------------+-------------------------------------
Reporter: RyanGlScott | Owner: (none)
Type: bug | Status: new
Priority: highest | Milestone: 8.2.1
Component: Compiler | Version: 8.1
Resolution: | Keywords:
Operating System: Unknown/Multiple | Architecture:
Type of failure: Compile-time | Unknown/Multiple
performance bug | Test Case:
Blocked By: | Blocking:
Related Tickets: #10800 | Differential Rev(s):
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by bgamari):
I setup a test environment to build just `tests/Utilities.hs`,
`tests/Boilerplater.hs`, and `tests/Tests/Vector.hs` with both GHC 8.3
(13131ce9165b4e5e5193dc381f6f3d021e53792f and 8.0.2. In addition to
building `Tests.Vector` as-is, I also built it enabling each of the
testcases in the `tests` list individually (that is, commenting out all
but one). The result is interesting: While compilation on `master` is
significantly faster than 8.0.2 in the individual cases, it is
significantly slower when all of the testcases are enabled. Clearly there
is something non-linear going on here. I'm on the case.
{{{
Building Utilities, Boilerplater, and Tests.Vector of vector
1d208ee9e3a252941ebd112e14e8cd5a982ac2bb
GHC 8.3 (13131ce9165b4e5e5193dc381f6f3d021e53792f
nothing: <
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Indeed in most phases of simplification the program with all testcases enabled is roughly three to five times larger in `master` than it was in `8.0.2`. After CorePrep, however, the program is nearly a factor of 8 larger, {{{ # 8.0.2 Result size of CorePrep = {terms: 118,005, types: 138,524, coercions: 50,256} # master Result size of CorePrep = {terms: 796,181, types: 907,739, coercions: 1,094,846, joins: 3,382/34,300} }}} It may also be interesting to note that if you sum the allocations of all of the testcases individually, you get something a bit higher than the allocations required to compile the "everything" case. I wonder if So, there are two (possibly related) questions here: 1. Why did the Core get so much larger from 8.0 to 8.2? 2. How was GHC 8.0 able to compile the "everything" case with less than a factor of two of the allocations of the individual cases, while 8.2 requires a factor of five? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Comparing the compilation of just the `Data.Vector.Vector (Bool)` with GHC 8.0.2 and 8.2 is also quite interesting: starting early in simplification we see that 8.2's core is typically less than half(!) the size of 8.0's (perhaps some goodness due to early inlining?). For instance, {{{ # 8.0.2 = {terms: 271,375, types: 552,012, coercions: 226,814} # 8.2 Result size of Simplifier iteration=1 = {terms: 105,610, types: 209,276, coercions: 135,191, joins: 299/5,710} }}} This remains the case throughout most of simplification; however, it seems that either Core Tidy or Core Prep cleans things up in 8.0, resulting in rather similar program sizes, {{{ # 8.0.2 Result size of CorePrep = {terms: 56,693, types: 73,297, coercions: 24,938} !!! CorePrep [Tests.Vector]: finished in 135.38 milliseconds, allocated 97.842 megabytes # 8.2 Result size of CorePrep = {terms: 69,730, types: 85,131, coercions: 125,647, joins: 248/2,453} !!! CorePrep [Tests.Vector]: finished in 132.14 milliseconds, allocated 106.584 megabytes }}} Seeing that the Core sizes are comparable when compiling a single testcase, question (2) is becoming quite interesting. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): As I suspected, there appears to be a large number of duplicate top-level bindings in the code that gets passed Core Tidy in 8.2. In particular, I count at least 25 copies of the same binding, {{{#!hs $s$wfoldlM_loop :: State# RealWorld-> [] Bool-> Int#-> MutableArray# RealWorld Bool-> Int#-> Int#-> (#,#) (TupleRep ([] RuntimeRep)) LiftedRep (State# RealWorld) (Vector Bool) {- Core Size{terms=209 types=244 cos=466 vbinds=7 jbinds=0} -} $s$wfoldlM_loop = ... }}} I'll need to work out why tomorrow. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): Can you describe how to reproduce this test case? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): You should be able to reproduce this with, {{{ $ git clone git://github.com/haskell/vector $ cd vector $ git checkout 1d208ee9e3a252941ebd112e14e8cd5a982ac2bb $ cabal install . --allow-newer=base,directory,process,haskeline,time,unix --disable-library-profiling --enable-test $ time ghc -no-link \ -i \ -idist/build/vector-tests-O2/vector-tests-O2-tmp \ -itests \ -idist/build/vector-tests-O2/autogen \ -idist/build/global-autogen \ -Idist/build/vector-tests-O2/autogen \ -Idist/build/global-autogen \ -Idist/build/vector-tests-O2/vector-tests-O2-tmp \ -optP-include \ -optPdist/build/vector-tests-O2/autogen/cabal_macros.h \ -XHaskell2010 \ -XCPP \ -XScopedTypeVariables \ -XPatternGuards \ -XMultiParamTypeClasses \ -XFlexibleContexts \ -XRank2Types \ -XTypeSynonymInstances \ -XTypeFamilies \ -XTemplateHaskell \ tests/Tests/Vector \ -O2 \ -Wall \ -fno-warn-orphans \ -fno-warn-missing-signatures \ -Wno-name-shadowing \ -Wno-unused-binds \ -Wno-unused-matches \ -Wno-unused-imports \ -ddump-to-file \ -ddump-simpl \ -dsuppress-idinfo \ -fforce-recomp \ -v \ +RTS -t 2>&1 | tee log }}} Then grep for `^\$s\$wfoldlM` in the `dump-simpl` output. You should see a number of top-level bindings of the type that I mention in comment:10. In my case they are all exactly of size, {{{ -- RHS size: {terms: 209, types: 244, coercions: 466, joins: 0/7} }}} and the right-hand sides are identical. It seems that the bindings in question are being excluded from CSE by `CSE.noCSE`. I suspect they may be join points, but I'm still trying to work out the particulars. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Indeed I've confirmed that the bindings in question are join points. I don't see any reason why top-level join points can't be CSE'd. I'll try fixing this. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Please excuse the nonsense in comment:13. My `trace` mislead me into believing that the bindings in question were join points when they were not (and now that I look again at this, I don't believe that such top- level join points could even arise). The problem is instead that the `not (isAlwaysActive (idInlineActivation id))` condition in `noCSE` fires for the bindings in question. This appears to be due to the `INLINE[0]` applied to these bindings by worker/wrapper. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:14 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): I'm still a bit boggled by how all these identical functions get created in the first place. I have not yet had a chance to gaze at the repro case. Do you understand why? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): So it turns out that the failure of CSE isn't actually a regression. I can observe the same repeated bindings produced by 8.0.2. Regardless, I think the CSE failure should probably be considered a bug. Afterall, worker/wrapper originally applies this inline pragma to ensure that the wrapper doesn't inline before any possible specialisations have had a chance to do so. If we CSE'd in this case we aren't going to hide any specialisation opportunities as the bindings we are CSE'ing are all identical and should therefore have the same available specialisations. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:16 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): simonpj, they the identical functions come into existence by inlining into numerous distinct call-sites of `foldlM` (which has `foldM_loop` let-bound in its unfolding). -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): OK, but all those call sites must be applied to the same combining- function argument (else the loop would be different in each case). So we just have lots of all to `foldlM k z`, for a particular `k` and `z`? Could we CSE those calls before inlining the `foldlM`? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:18 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari):
Could we CSE those calls before inlining the foldlM?
Perhaps, but It seems like what is happening is this, 1. The specialiser creates an `INLINE` specialisation, {{{ $sunstream :: Bundle Vector Int -> New Vector Int }}} which has `foldlM` (and consequently `foldlM_loop`) let-bound within it 2. FloatOut gets run, but since we are in an `InlineCtxt` we don't float the `foldlM_loop`s out to top-level where CSE might be able to consolidate them (incidentally, is `Note [FloatOut inside INLINE]` in `SetLevels` still correct? It doesn't seem to be referenced from anywhere) 3. The specialisation is inlined into dozens of callsites 4. We do lots of expensive, redundant simplification 5. Eventually we do CSE, but even then we don't eliminate the redundant top-level `foldlM_loop`s for the reason discussed in comment:16 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari):
(incidentally, is Note [FloatOut inside INLINE] in SetLevels still correct? It doesn't seem to be referenced from anywhere)
It actually appears that the logic associated with this note was refactored in, {{{ commit c0fe534fbf3d3e30a6f35b355da86e6f18601348 Author: simonpj@microsoft.com <unknown> Date: Tue Apr 22 11:50:03 2008 +0000 Fix a long-standing bug in FloatOut }}} and eventually removed in, {{{ commit 72462499b891d5779c19f3bda03f96e24f9554ae Author: simonpj@microsoft.com <unknown> Date: Thu Oct 29 14:30:51 2009 +0000 The Big INLINE Patch: totally reorganise way that INLINE pragmas work }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:20 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj):
So it turns out that the failure of CSE isn't actually a regression. I can observe the same repeated bindings produced by 8.0.2.
But nevertheless, regardless of CSE, things got worse between 8.0 and 8.2 and we do not yet know why. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: (none) Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): ToDo: 1. Vector library author: do we really need to INLINE `unstream` and friends, or would INLINABLE do? 2. Why is 8.2 worse than 8.0? 3. Maybe we could do a better job of CSE as discussed in #13589. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:22 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * owner: (none) => bgamari -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:23 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by George): with ghc 8.2.1-rc2 on a Mac, succeeds in less than 4gb , watching Activity Monitor, Memory max usage seems between 3.5 and 4gb: {{{ ulimit -v 4000000; cabal test ... Preprocessing test suite 'vector-tests-O2' for vector-0.12.0.1... ... [4 of 7] Compiling Tests.Vector ( tests/Tests/Vector.hs, dist/build /vector-tests-O2/vector-tests-O2-tmp/Tests/Vector.o ) ... tests/Tests/Vector.hs:438:5: warning: [-Wname-shadowing] This binding for ‘limitUnfolds’ shadows the existing binding imported from ‘Utilities’ at tests/Tests/Vector.hs:5:1-24 (and originally defined at tests/Utilities.hs:347:1-12) | 438 | limitUnfolds f (theirs, ours) | ^^^^^^^^^^^^ [5 of 7] Compiling Tests.Move ( tests/Tests/Move.hs, dist/build /vector-tests-O2/vector-tests-O2-tmp/Tests/Move.o ) ... [7 of 7] Compiling Main ( tests/Main.hs, dist/build/vector- tests-O2/vector-tests-O2-tmp/Main.o ) Linking dist/build/vector-tests-O2/vector-tests-O2 ... Running 2 test suites... Test suite vector-tests-O0: RUNNING... Test suite vector-tests-O0: PASS Test suite logged to: dist/test/vector-0.12.0.1-vector-tests-O0.log Test suite vector-tests-O2: RUNNING... Test suite vector-tests-O2: PASS Test suite logged to: dist/test/vector-0.12.0.1-vector-tests-O2.log 2 of 2 test suites (2 of 2 test cases) passed. bash-3.2$ ghc --version The Glorious Glasgow Haskell Compilation System, version 8.2.0.20170507 }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:24 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by RyanGlScott): 4 GB was a somewhat arbitrary number I picked. On my machine, compiling the tests on GHC 8.0.2 took about 1.6 GB, whereas with GHC 8.2.1, it took about 6.7 GB. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:25 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by RyanGlScott): I've finally managed to identify the commit that caused this memory spike: 1c4a39d3a8d36803382792ff78b4709794358883 (Prioritise class-level equality costraints). -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:26 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by RyanGlScott): I've also confirmed that reverting 1c4a39d3a8d36803382792ff78b4709794358883 from the current master (d39a3409acd3c40fb018ec1c114f15d3ecef6ef9) makes GHC able to compile the `vector` test suite again, so that is a viable workaround. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:27 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Great work Ryan! That is a great hint. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:28 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): It is a bit surprising to hear that this patch is the culprit, however, as it appears to be solely a typechecker change. Perhaps there is some change in the generated Core which then triggers some latent simplifier bug. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:29 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2
-------------------------------------+-------------------------------------
Reporter: RyanGlScott | Owner: bgamari
Type: bug | Status: new
Priority: highest | Milestone: 8.2.1
Component: Compiler | Version: 8.1
Resolution: | Keywords:
Operating System: Unknown/Multiple | Architecture:
Type of failure: Compile-time | Unknown/Multiple
performance bug | Test Case:
Blocked By: | Blocking:
Related Tickets: #10800 | Differential Rev(s):
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by dfeuer):
FWIW, this is what I see on HEAD right now, compiling with
{{{
~/src/ghc-clean-june28/inplace/bin/ghc-stage2 -no-link -fbuilding-cabal-
package -static -dynamic-too -dynosuf dyn_o -dynhisuf dyn_hi -outputdir
dist/build/vector-tests-O2/vector-tests-O2-tmp -odir dist/build/vector-
tests-O2/vector-tests-O2-tmp -hidir dist/build/vector-tests-O2/vector-
tests-O2-tmp -stubdir dist/build/vector-tests-O2/vector-tests-O2-tmp -i
-idist/build/vector-tests-O2/vector-tests-O2-tmp -itests
-idist/build/autogen -Idist/build/autogen -Idist/build/vector-tests-O2
/vector-tests-O2-tmp -optP-include -optPdist/build/autogen/cabal_macros.h
-hide-all-packages -package-db dist/package.conf.inplace -package
HUnit-1.6.0.0 -package QuickCheck-2.9.2 -package base -package random-1.1
-package template-haskell -package test-framework-0.8.1.1 -package test-
framework-hunit-0.3.0.2 -package test-framework-quickcheck2-0.3.0.3
-package transformers-0.5.2.0 -package vector-0.12.0.1 -XHaskell2010 -XCPP
-XScopedTypeVariables -XPatternGuards -XMultiParamTypeClasses
-XFlexibleContexts -XRank2Types -XTypeSynonymInstances -XTypeFamilies
-XTemplateHaskell tests/Tests/Vector.hs -O2 -fno-warn-orphans -fno-warn-
missing-signatures -Rghc-timing
}}}
{{{
DV Bool:
<
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): David will summarise by saying * Compile this module X with these flags * Compiler allocations (a) pre offending patch (b) post offending patch (c) HEAD * Tidy Core sizes for (a)-(c); i.e. is it slower because we are compiling more code or because we are taking long to compile it? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:31 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by dfeuer): * Attachment "Vector.hs" added. A slightly smaller example -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by dfeuer): * Attachment "Vector.hs" added. Reduced a little more -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by dfeuer): Starting with `vector` 1d208ee9e3a252941ebd112e14e8cd5a982ac2bb, replace `tests/Test/Vector.hs` with the attached file. Compile using {{{ -no-link -fbuilding-cabal-package -static -dynamic-too -dynosuf dyn_o \ -dynhisuf dyn_hi -outputdir dist/build/vector-tests-O2/vector-tests-O2-tmp \ -odir dist/build/vector-tests-O2/vector-tests-O2-tmp -hidir \ dist/build/vector-tests-O2/vector-tests-O2-tmp -stubdir \ dist/build/vector-tests-O2/vector-tests-O2-tmp -i \ -idist/build/vector-tests-O2/vector-tests-O2-tmp -itests \ -idist/build/autogen -Idist/build/autogen \ -Idist/build/vector-tests-O2/vector-tests-O2-tmp \ -optP-include -optPdist/build/autogen/cabal_macros.h -hide-all-packages \ -package-db dist/package.conf.inplace -package HUnit-1.6.0.0 \ -package QuickCheck-2.9.2 -package base -package random-1.1 -package \ template-haskell -package test-framework-0.8.1.1 -package \ test-framework-hunit-0.3.0.2 -package test-framework-quickcheck2-0.3.0.3 \ -package transformers-0.5.2.0 -package vector-0.12.0.1 -XHaskell2010 \ -XCPP -XScopedTypeVariables -XPatternGuards -XMultiParamTypeClasses \ -XFlexibleContexts -XRank2Types -XTypeSynonymInstances -XTypeFamilies \ -XTemplateHaskell tests/Tests/Vector.hs -O2 -fno-warn-orphans \ -fno-warn-missing-signatures -Rghc-timing -v -ddump-simpl -ddump-to-file }}} In the below, the "offending patch" is 1c4a39d3a8d36803382792ff78b4709794358883 and "recent HEAD" is 7097f94df0c6667be2343306ffcda152fa22abcf. Allocations: 1. Before the offending patch: 13,445,819,064 2. After the offending patch: 135,653,550,528 (10x worse) 3. Recent HEAD: 71,354,312,480 (~5x worse) Tidy core size (terms) 1. Before offending: 20,934 2. After offending: 170,128 3. Recent HEAD: 170,023 Maximum reported core size (terms) before Tidy Core: 1. Before offending: 45,213 2. After offending: 1,358,264 3. Recent HEAD: 503,803 So we are clearly generating ''far'' more code, and while we eliminate a lot of it, we're still left with an awful lot. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:32 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by dfeuer): * Attachment "ds-diff" added. dump-ds diff for offending patch -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by dfeuer): I've attached a diff of the results of `-ddump-ds -dsuppress-uniques` before and after the offending patch. The change looks pretty small, but something is blowing everything up. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:33 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by dfeuer): * Attachment "ds.diff" added. Proper extension on the diff -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by dfeuer): * Attachment "Smaller-Vector.hs" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by dfeuer): * Attachment "smaller-ds.diff" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by dfeuer): I cut down the test case some more, to `Smaller-Vector.hs`, with desugar dump in `smaller-ds.diff`. The relative performance contrast isn't ''quite'' as strong, but it's still almost five times as bad in HEAD as before the badness started. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:34 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.2.2 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * milestone: 8.2.1 => 8.2.2 Comment: We are going to punt this to 8.2.2. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:35 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * priority: highest => high * milestone: 8.2.2 => 8.4.1 Comment: Actually, perhaps 8.4. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:36 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2
-------------------------------------+-------------------------------------
Reporter: RyanGlScott | Owner: bgamari
Type: bug | Status: new
Priority: high | Milestone: 8.4.1
Component: Compiler | Version: 8.1
Resolution: | Keywords:
Operating System: Unknown/Multiple | Architecture:
Type of failure: Compile-time | Unknown/Multiple
performance bug | Test Case:
Blocked By: | Blocking:
Related Tickets: #10800 | Differential Rev(s):
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by George):
Given the difficulty of fixing compiler bugs of excessive memory use, does
it make sense to introduce a test to prevent future regressions:
bash -c 'ulimit -v
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by George): * cc: george (removed) * cc: George (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:38 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Well, we have a testsuite which includes a variety of performance tests. However, they tend to be "microbenchmarks" are don't approach the scale of `vector`. Unfortunately, these tests IMHO already require far more effort to maintain than they give us benefit due to the need for manual bumping of thresholds. The problem is summarized in #12758 (which jared-w is working on fixing this summer). I do think there is room for more compiler performance tests, but I think they should generally be something that we monitor, not something that we actively need to bump thresholds on. As far as using the `vector` testsuite for such a testcase is concerned, I'd skeptical of including it wholesale. Rather, I think we would be better off building a minimized version of `vector`'s testcase (which will hopefully fall out naturally as a result of working on this ticket) and including that. Testcases with lots of dependencies are generally problematic as you get breakage due to interface changes, spurious performance fluctuations due to changes in dependencies, and the pain of having to track down where in your testcase GHC is regressing. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:39 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by lippling): (I'm not exactly sure if this is the right ticket) I tried to compile one of our server applications with 8.2.1 (which compiles fine with 8.0.2).\\\\ The compilation runs smooth, but when it reaches a specific file, the RAM usage goes up to > 20GB pretty fast on my 16GB machine and the GHC process gets terminated with an stack overflow error. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:40 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
(I'm not exactly sure if this is the right ticket)
I tried to compile one of our server applications with 8.2.1 (which compiles fine with 8.0.2).\\\\ The compilation runs smooth, but when it reaches a specific file, the RAM usage goes up to > 20GB pretty fast on my 16GB machine and the GHC
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): Replying to [comment:40 lippling]: process gets terminated with an stack overflow error. If you can take the time to reduce your program to a couple of shareable modules it would be appreciated if you could open a new ticket with your repro. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:41 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by lippling): Ok, I created https://ghc.haskell.org/trac/ghc/ticket/14163#ticket -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:42 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.4.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): This won't be fixed for 8.4.1. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:43 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.6.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * milestone: 8.4.1 => 8.6.1 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:44 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.6.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by George): Replying to [comment:24 George]: this seems to have regressed with 8.4.1, original test case is now using 10-11 gb on my mac -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:45 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.6.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by George): As mentioned in the description the problem happens when compiling one particular file, it would be nice if somebody could reduce the test case to that so we could profile the compiler compiling that one file. This issues is still reproducible in ghc 8.4.3 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:46 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.8.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by harendra): It looks like I am seeing this same bug on GHC 8.2.2. This is again in a streaming library with stream fusion like vector and I am seeing the problem specifically with the "filter" code in the library, though it does not always happen, only in some cases. I have two branches in my repo that reproduce the problem: * See branch https://github.com/composewell/streamly/tree/ghc-8.2.2-bug . The last commit on this branch https://github.com/composewell/streamly/commit/8f08248eba6702159f7bc3fe99e0c... disables the culprit code. * The second branch is https://github.com/composewell/streamly/tree/ghc-8.2.2-bug2 . The last commit on this branch disables the culprit code. The "filter" API code that is being used in these cases is defined in https://github.com/composewell/streamly/blob/ghc-8.2.2-bug2/src/Streamly/Str... . The code goes like this: {{{ {-# INLINE_NORMAL filterM #-} filterM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a filterM f (Stream step state) = Stream step' state where {-# INLINE_LATE step' #-} step' gst st = do r <- step (rstState gst) st case r of Yield x s -> do b <- f x if b then return $ Yield x s else step' gst s Stop -> return $ Stop {-# INLINE filter #-} filter :: Monad m => (a -> Bool) -> Stream m a -> Stream m a filter f = filterM (return . f) }}} This is very much like the vector code, except that there is no Skip constructor. I was originally thinking that it may have something to do with the join point optimization. I hope this will shed some light on the issue. I am not seeing the issue with GHC-8.4. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:48 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.8.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj): harendra: is your issue also fixed in 8.6? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:49 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.8.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): If comment: 49 is just an 8.2 issue and isn't reproducible on 8.4 or 8.6 then we should likely consider it to be a different (fixed!) bug than the original one that prompted this ticket. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:50 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.8.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by George): * Attachment "compile.out" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.8.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by George): The original test still reproduces with ghc 8.6.1. In particular, bash -c 'ulimit -v 4000000; cabal test' fails with "ghc: out of memory." The output of David's compile above with Smaller-Vector.hs is in the attached file, compile.out, just above -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:51 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.8.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by tuxArg): Hi, It seems this bug is affecting me. I'm benchmarking my code (not test suites) with stack. The code does many zipWith, map and foldl over storable vectors with double values. Here are some rts stats comparison between 8.0.2 (last stack ok, lts-9.21) and 8.2.2 (lts-10.0): {{{ ghc 8.0.2: 66,593,737,608 bytes allocated in the heap 930,040,096 bytes copied during GC 15 MB total memory in use (0 MB lost due to fragmentation) Tot time (elapsed) Avg pause Max pause Gen 0 10101 colls, 0 par 1.424s 1.548s 0.0002s 0.0014s Gen 1 879 colls, 0 par 0.408s 0.377s 0.0004s 0.0018s Total time 74.408s ( 75.442s elapsed) %GC time 2.5% (2.6% elapsed) Alloc rate 917,572,442 bytes per MUT second Productivity 97.5% of total user, 97.4% of total elapsed ghc 8.2.2: 317,181,489,136 bytes allocated in the heap 1,806,653,248 bytes copied during GC 15 MB total memory in use (0 MB lost due to fragmentation) Tot time (elapsed) Avg pause Max pause Gen 0 68766 colls, 0 par 3.700s 3.448s 0.0001s 0.0024s Gen 1 1542 colls, 0 par 0.164s 0.157s 0.0001s 0.0009s Total time 118.112s (119.443s elapsed) %GC time 3.3% (3.0% elapsed) Alloc rate 2,776,254,193 bytes per MUT second Productivity 96.7% of total user, 97.0% of total elapsed }}} Is there any known workaround?. This issue is not allowing me to upgrade from ghc 8.0.2. Thanks. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:52 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.8.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by osa1): * cc: osa1 (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:53 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: high | Milestone: 8.8.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by gnezdo): * cc: gnezdo (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:54 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13535: vector test suite uses excessive memory on GHC 8.2 -------------------------------------+------------------------------------- Reporter: RyanGlScott | Owner: bgamari Type: bug | Status: new Priority: highest | Milestone: 8.10.1 Component: Compiler | Version: 8.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: #10800 | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * priority: high => highest * milestone: 8.8.1 => 8.10.1 Comment: Given the recent activity on this I am going to bump its priority again. We should try to work this out once and for all. However, it doesn't seem likely that this will happen on a 8.8-compatible timescale. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13535#comment:55 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC