[GHC] #11380: `cabal repl` exhausts memory
#11380: `cabal repl` exhausts memory -------------------------------------+------------------------------------- Reporter: kennethb | Owner: Type: bug | Status: new Priority: high | Milestone: Component: Compiler | Version: 7.10.2 Keywords: | Operating System: Unknown/Multiple Architecture: x86_64 | Type of failure: Compile-time (amd64) | crash Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- I have a project that depends on happy/alex to generate parsers. (https://github.com/TOSPIO/pyn) Running `cabal repl` on my laptop (Gentoo Linux amd64) with 16GiB memory ended up eating up all memory and eventually resulted in kernel panic. `cabal build` works fine. Problem arises when compiling the giant happy- generated dist/build/Language/Python/Parser/Parse.hs file also `cabal repl --ghc-options=-fobject-code` works fine. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11380 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#11380: `cabal repl` exhausts memory -------------------------------------+------------------------------------- Reporter: kennethb | Owner: Type: bug | Status: new Priority: high | Milestone: Component: Compiler | Version: 7.10.2 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 Type of failure: Compile-time | (amd64) crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by kennethb): * os: Unknown/Multiple => Linux Old description:
I have a project that depends on happy/alex to generate parsers. (https://github.com/TOSPIO/pyn)
Running `cabal repl` on my laptop (Gentoo Linux amd64) with 16GiB memory ended up eating up all memory and eventually resulted in kernel panic.
`cabal build` works fine. Problem arises when compiling the giant happy- generated dist/build/Language/Python/Parser/Parse.hs file
also `cabal repl --ghc-options=-fobject-code` works fine.
New description: I have a project that depends on happy/alex to generate parsers. (https://github.com/TOSPIO/pyn) Running `cabal repl` on my laptop with 16GiB memory ended up eating up all memory and eventually resulted in kernel panic. `cabal build` works fine. Problem arises when compiling the giant happy- generated dist/build/Language/Python/Parser/Parse.hs file also `cabal repl --ghc-options=-fobject-code` works fine. -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11380#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#11380: `cabal repl` exhausts memory -------------------------------------+------------------------------------- Reporter: kennethb | Owner: Type: bug | Status: new Priority: high | Milestone: Component: Compiler | Version: 7.10.3 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 Type of failure: Compile-time | (amd64) crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by kennethb): * version: 7.10.2 => 7.10.3 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11380#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#11380: `cabal repl` exhausts memory -------------------------------------+------------------------------------- Reporter: kennethb | Owner: Type: bug | Status: new Priority: high | Milestone: Component: Compiler | Version: 7.10.3 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 Type of failure: Compile-time | (amd64) crash | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Old description:
I have a project that depends on happy/alex to generate parsers. (https://github.com/TOSPIO/pyn)
Running `cabal repl` on my laptop with 16GiB memory ended up eating up all memory and eventually resulted in kernel panic.
`cabal build` works fine. Problem arises when compiling the giant happy- generated dist/build/Language/Python/Parser/Parse.hs file
also `cabal repl --ghc-options=-fobject-code` works fine.
New description: I have a project that depends on happy/alex to generate parsers. (https://github.com/TOSPIO/pyn) Running `cabal repl` on my laptop with 16GiB memory ended up eating up all memory and eventually resulted in kernel panic. Problem arises when compiling the giant happy-generated dist/build/Language/Python/Parser/Parse.hs file. The memory usage is flat (~1.5G) within the first 2 mins, and the soars up to over 10GB within seconds. `cabal build` and `cabal repl --ghc-options=-fobject-code` works fine. -- Comment (by kennethb): I noticed that -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11380#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#11380: Compiling a 10.000 line file exhausts memory -------------------------------------+------------------------------------- Reporter: kennethb | Owner: Type: bug | Status: new Priority: high | Milestone: Component: Compiler | Version: 7.10.3 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 Type of failure: Compile-time | (amd64) performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by thomie): * failure: Compile-time crash => Compile-time performance bug Comment: Compiling (instead of interpreting) the generated Parser.hs with ghc-7.10.3 (wihout optimizations) doesn't complete either, when only 3GB of RAM is available (tested with `ulimit -v 3000000`). The problems seem to begin in the desugaring phase: {{{ *** Parser: *** Renamer/typechecker: *** Desugar: ghc: out of memory (requested 2097152 bytes) }}} The generated parser is huge (10.000 lines). But it is about the same size as GHC's own parser, which GHC //is// able to compile with reasonable memory. I tried making the Parser smaller a bit, but didn't find any obvious rule that might be triggering the problem. Could you try with other version of GHC (say 7.8.4 and 8.0.1)? Is it a regression, or maybe already fixed? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11380#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#11380: Compiling a 10.000 line file exhausts memory -------------------------------------+------------------------------------- Reporter: kennethb | Owner: Type: bug | Status: new Priority: high | Milestone: Component: Compiler | Version: 7.10.3 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 Type of failure: Compile-time | (amd64) performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by jstolarek): kennethb, what happens when you try to compile your parser with `-O0` option (ie. without optimizations)? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11380#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#11380: Compiling a 10.000 line file exhausts memory -------------------------------------+------------------------------------- Reporter: kennethb | Owner: Type: bug | Status: new Priority: high | Milestone: Component: Compiler | Version: 7.10.3 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 Type of failure: Compile-time | (amd64) performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by thomie): `happy` has a `-a` flag: {{{ -a --array generate an array-based parser }}} Cabal calls `happy -agc` by default, but I was using just `happy` in comment:4. It seems to influence the results when trying to compile `Parser.hs` within 3GB RAM. || ghc-7.10.3 Parser.hs ||= happy =||= happy -a =|| || -O0 || not ok || ok || || -O1 || ok || ok || @jstolek: compiling the non-array based parser //without// optimizations fails. Very strange. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11380#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#11380: Compiling a 10.000 line file exhausts memory -------------------------------------+------------------------------------- Reporter: kennethb | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.3 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 Type of failure: Compile-time | (amd64) performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by rwbarton): * priority: high => normal Comment: Let's stick to the array-based parser since that's what the original issue was about. Then there's no problem compiling, either with or without optimizations. The problem is only with ghci for some reason. Trying with `-v`, there seems to be some major early difference when building with ghc vs. ghci. {{{ rwbarton@morphism:/tmp/pyn$ ghc-7.10.1 -fforce-recomp -c dist/build/Language/Python/Pyn/Parser/Parser.hs -O0 -ilib -idist/build -v -this-package-key pyn_4LeYrll2NJX6hY3DO8vvbp Glasgow Haskell Compiler, Version 7.10.1, stage 2 booted by GHC version 7.8.4 [...] *** Desugar: Result size of Desugar (after optimization) = {terms: 214,221, types: 3,864,905, coercions: 74,929} [... everything is fine] }}} {{{ rwbarton@morphism:/tmp/pyn$ ghci-7.10.1 dist/build/Language/Python/Pyn/Parser/Parser.hs -ilib -idist/build -v -this-package-key pyn_4LeYrll2NJX6hY3DO8vvbp [...] [10 of 10] Compiling Language.Python.Pyn.Parser.Parser ( dist/build/Language/Python/Pyn/Parser/Parser.hs, interpreted ) *** Parser: *** Renamer/typechecker: *** Desugar: Result size of Desugar (after optimization) = {terms: 220,453, types: 414,293,970, coercions: 75,337} *** Simplifier: Result size of Simplifier iteration=1 = {terms: 220,516, types: 419,756,722, coercions: 577,145} Result size of Simplifier iteration=2 = {terms: 220,516, types: 419,756,722, coercions: 370,154} Result size of Simplifier = {terms: 220,514, types: 419,756,719, coercions: 370,206} *** Tidy Core: Result size of Tidy Core^CInterrupted. [here memory usage exploded] }}} What would cause this huge difference? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11380#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#11380: Compiling a 10.000 line file exhausts memory -------------------------------------+------------------------------------- Reporter: kennethb | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.10.3 Resolution: | Keywords: Operating System: Linux | Architecture: x86_64 Type of failure: Compile-time | (amd64) performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by AndreasK): This regressed farther in 8.2.2 and 8.0.2 (at -O1) Compiling the project in GHC-8.2.2 takes forever (~10 Hours on an i7-6700). Ram usage for compile or ghci didn't go over 3GB in the first 10 Minutes but I stopped watching it afterwards. So the compile time of ~10h could have been a result of my machine running out of ram. With 7.10.3 compiling takes less than two Minutes. GHC-8.0.2 (stackage lts-9.20) probably has the same behaviour as 8.2.2 but I didn't wait around for it to finish. Using the same lts with ghc-7.10.3 works fine. Logs for build/ghci with 8.2.2: [https://github.com/AndreasPK/pyn/tree/f77f9454aa424545b2ce8bfb460d8d275edda9... repo for repro] {{{ Compiled (-O1) [17 of 20] Compiling Language.Python.Pyn.Parser.Parser ( .stack- work\dist\5c8418a7\build\Language\Python\Pyn\Parser\Parser.hs, .stack- work\dist\5c8418a7\build\Language\Python\Pyn\Parser\Parser.o ) *** Checking old interface for Language.Python.Pyn.Lexer (use -ddump-hi- diffs for more details): *** Deleting temp files: Deleting: compile: input file C:\Users\Andi\AppData\Local\Temp\ghc81932_0\ghc_7.hscpp *** Checking old interface for Language.Python.Pyn.Parser.Parser (use -ddump-hi-diffs for more details): *** Parser [Language.Python.Pyn.Parser.Parser]: !!! Parser [Language.Python.Pyn.Parser.Parser]: finished in 937.50 milliseconds, allocated 2647.317 megabytes *** Renamer/typechecker [Language.Python.Pyn.Parser.Parser]: !!! Renamer/typechecker [Language.Python.Pyn.Parser.Parser]: finished in 35763203.12 milliseconds, allocated 109977425.131 megabytes *** Desugar [Language.Python.Pyn.Parser.Parser]: Result size of Desugar (after optimization) = {terms: 83,551, types: 2,023,928, coercions: 76,061, joins: 1/136} Interpreter: [17 of 21] Compiling Language.Python.Pyn.Parser.Parser ( C:\test\pyn \.stack-work\dist\5c8418a7\build\Language\Python\Pyn\Parser\Parser.hs, interpreted ) *** Parser [Language.Python.Pyn.Parser.Parser]: !!! Parser [Language.Python.Pyn.Parser.Parser]: finished in 890.62 milliseconds, allocated 2646.366 megabytes *** Renamer/typechecker [Language.Python.Pyn.Parser.Parser]: !!! Renamer/typechecker [Language.Python.Pyn.Parser.Parser]: finished in 35747375.00 milliseconds, allocated 109978920.259 megabytes *** Desugar [Language.Python.Pyn.Parser.Parser]: Result size of Desugar (after optimization) = {terms: 217,920, types: 3,341,879, coercions: 76,475, joins: 0/653} }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/11380#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC