[GHC] #13639: Skylighting package compilation is glacial
#13639: Skylighting package compilation is glacial -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.4.1 Component: Compiler | Version: 8.0.1 Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: Compile-time Unknown/Multiple | performance bug Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- Compiling the `skylighting` package in profiled and non-profiled confihratiine takes nearly ten minutes. I haven't looked at the code, but I can't imagine a syntax highlighting package would need to take this long. Investigate. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13639 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13639: Skylighting package compilation is glacial -------------------------------------+------------------------------------- Reporter: bgamari | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.4.1 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Description changed by bgamari: @@ -2,1 +2,1 @@ - confihratiine takes nearly ten minutes. I haven't looked at the code, but + configuration takes nearly ten minutes. I haven't looked at the code, but New description: Compiling the `skylighting` package in profiled and non-profiled configuration takes nearly ten minutes. I haven't looked at the code, but I can't imagine a syntax highlighting package would need to take this long. Investigate. -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13639#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13639: Skylighting package compilation is glacial -------------------------------------+------------------------------------- Reporter: bgamari | Owner: dfeuer Type: bug | Status: new Priority: normal | Milestone: 8.4.1 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by dfeuer): * owner: (none) => dfeuer Comment: I'll take a look and see if I can narrow it down to a module or two. A brief skim of the code didn't show anything too weird, so I'm guessing we're getting a lot of inlining or specialization from dependencies. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13639#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13639: Skylighting package compilation is glacial -------------------------------------+------------------------------------- Reporter: bgamari | Owner: dfeuer Type: bug | Status: new Priority: normal | Milestone: 8.4.1 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by dfeuer): * cc: mpickering (added) Comment: I'm just getting back to looking at this. Since we don't have a record of the problematic `skylighting` version, let's go with the guess that this is the issue reported as [https://github.com/jgm/skylighting/issues/7 skylighting issue #7]. That is, that `skylighting-0.1.1.5` compiles slowly. There is further discussion of the problem in [https://github.com/jgm/texmath/issues/96 texmath issue #96]. mpickering was involved in those discussions; I don't know how much he knows/remembers. Their conclusion was that very large static data can take a long time to compile, for some reason. This whole thing sounds very familiar somehow; wasn't there some recentish work on handling ticks better for long list literals or some such? The package worked around this problem in [https://github.com/jgm/skylighting/commit/851bdd1e210ddecb00ceeffcabd9bf1fdc... Skylighting.Syntax.*: use string representation of the Syntax] which simply used `read` to read a `String` representation of the data. Later, they [https://github.com/jgm/skylighting/commit/90ae0d0c23b7f8b218ff01152a912e3742... switched to Binary]. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13639#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13639: Skylighting package compilation is glacial -------------------------------------+------------------------------------- Reporter: bgamari | Owner: dfeuer Type: bug | Status: new Priority: normal | Milestone: 8.4.1 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by dfeuer): In a recent HEAD ( 4700baaf8f9bf3c44a53a595d840c7c14cfd6f98 ): The "nearly ten minutes" seems to be the "user" time in a parallel build, for which the real time is around 3 minutes. If cabal and GHC are each passed `-j1`, the user time drops to around five minutes. So one issue seems to be some inefficiency in parallel builds. Running single-threaded, I don't see any big difference between the time it takes to build the unprofiled and profiled versions (although of course building both takes about twice as long as building just the unprofiled). So I don't really think profiling has anything to do with the issue, at least for HEAD. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13639#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13639: Skylighting package compilation is glacial -------------------------------------+------------------------------------- Reporter: bgamari | Owner: dfeuer Type: bug | Status: new Priority: normal | Milestone: 8.4.1 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by niteria): This package has 123 modules which contain nothing else than a giant multi-kilobyte `String`. See https://hackage.haskell.org/package/skylighting-0.3.3.1/src/src/Skylighting/... for example. With `-g` (DWARF) on `ghc-8.0.2` it appears to lead to ~40m compile times. I have a strong suspicion that we end up generating debugging information for each of these `(:)` cells, which is bound to be slow with 4 million of these total and the nesting levels of 10k. I may be wrong because #11095 was supposed to improve it and I'm running with https://git.haskell.org/ghc.git/commitdiff/29122312cc7b8f9890eb53f92d76ecdd8... already. I will try to confirm if the problem persists on HEAD. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13639#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13639: Skylighting package compilation is glacial -------------------------------------+------------------------------------- Reporter: bgamari | Owner: dfeuer Type: bug | Status: new Priority: normal | Milestone: 8.4.1 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by niteria): Ignore what I said above, it's about a different version of skylighting which doesn't appear to have compile time issues. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13639#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13639: Skylighting package compilation is glacial -------------------------------------+------------------------------------- Reporter: bgamari | Owner: dfeuer Type: bug | Status: new Priority: normal | Milestone: 8.4.1 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by niteria): This bisects to a6e13d502ef46de854ec1babcd764ccce68c95e3, meaning that commit fixes the underlying problem. I cherry-picked it onto our `ghc-8.0.2-facebook` branch and the compile times are reasonable (~2min as opposed to >30m). I minified the original code to Skylighting.Syntax.Php.hs file that I'm attaching. It's boils down to a large `[Text]` list. With a commit before the fix there's a big blow up in the term size after one of the simplifier passes. After the fix the term sizes look reasonable. Relevant part of `-ddump-show-phases` (this is the released GHC 8.0.2, but the numbers are the the same before the fix): {{{ !!! Parser [Skylighting.Syntax.Php]: finished in 4556.00 milliseconds, allocated 87.667 megabytes [99/1907] *** Renamer/typechecker [Skylighting.Syntax.Php]: !!! Renamer/typechecker [Skylighting.Syntax.Php]: finished in 27009.00 milliseconds, allocated 241.752 megabytes *** Desugar [Skylighting.Syntax.Php]: Result size of Desugar (after optimization) = {terms: 16,197, types: 6,490, coercions: 0} !!! Desugar [Skylighting.Syntax.Php]: finished in 2024.00 milliseconds, allocated 22.967 megabytes *** Simplifier [Skylighting.Syntax.Php]: Result size of Simplifier iteration=1 = {terms: 22,671, types: 19,426, coercions: 0} Result size of Simplifier = {terms: 22,671, types: 19,426, coercions: 0} !!! Simplifier [Skylighting.Syntax.Php]: finished in 14080.00 milliseconds, allocated 150.808 megabytes *** Specialise [Skylighting.Syntax.Php]: Result size of Specialise = {terms: 22,678, types: 19,437, coercions: 2} !!! Specialise [Skylighting.Syntax.Php]: finished in 5593.00 milliseconds, allocated 32.625 megabytes *** Float out(FOS {Lam = Just 0, Consts = True, OverSatApps = False}) [Skylighting.Syntax.Php]: Result size of Float out(FOS {Lam = Just 0, Consts = True, OverSatApps = False}) = {terms: 48,542, types: 58,233, coercions: 2} !!! Float out(FOS {Lam = Just 0, Consts = True, OverSatApps = False}) [Skylighting.Syntax.Php]: finished in 8951.00 milliseconds, allocated 126.301 megabytes *** Simplifier [Skylighting.Syntax.Php]: Result size of Simplifier iteration=1 = {terms: 45,307, types: 55,000, coercions: 0} Result size of Simplifier = {terms: 45,307, types: 55,000, coercions: 0} !!! Simplifier [Skylighting.Syntax.Php]: finished in 36008.00 milliseconds, allocated 394.685 megabytes *** Simplifier [Skylighting.Syntax.Php]: Result size of Simplifier iteration=1 = {terms: 55,006, types: 55,000, coercions: 0} Result size of Simplifier = {terms: 42,074, types: 22,670, coercions: 0} !!! Simplifier [Skylighting.Syntax.Php]: finished in 44152.00 milliseconds, allocated 564.940 megabytes *** Simplifier [Skylighting.Syntax.Php]: Result size of Simplifier iteration=1 = {terms: 1,060,469, types: 775,959, coercions: 193,980} Result size of Simplifier iteration=2 = {terms: 944,081, types: 824,454, coercions: 171,349} Result size of Simplifier iteration=3 = {terms: 827,693, types: 624,008, coercions: 38,796} Result size of Simplifier iteration=4 }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13639#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13639: Skylighting package compilation is glacial -------------------------------------+------------------------------------- Reporter: bgamari | Owner: dfeuer Type: bug | Status: new Priority: normal | Milestone: 8.4.1 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by niteria): * Attachment "Skylighting.Syntax.Php.hs" added. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13639 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13639: Skylighting package compilation is glacial -------------------------------------+------------------------------------- Reporter: bgamari | Owner: dfeuer Type: bug | Status: new Priority: normal | Milestone: 8.4.1 Component: Compiler | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonpj):
meaning that commit fixes the underlying problem.
So can we close this ticket? Or is there still a problem? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13639#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#13639: Skylighting package compilation is glacial -------------------------------------+------------------------------------- Reporter: bgamari | Owner: dfeuer Type: bug | Status: closed Priority: normal | Milestone: 8.6.1 Component: Compiler | Version: 8.0.1 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Compile-time | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => closed * resolution: => fixed Comment: I believe this is resolved. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/13639#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC