[GHC] #11783: Very large slowdown when using parallel garbage collector
#11783: Very large slowdown when using parallel garbage collector -------------------------------------+------------------------------------- Reporter: luispedro | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Runtime | Version: 7.10.3 System | Keywords: performance, | Operating System: Unknown/Multiple garbage collector | Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- As part of debugging some performance issues on an application I am writing, I concluded that the issue is in the parallel GC implemented in the GHC RTS. I extracted the code attached to make a self-contained use- case, but in my system the code runs in 16s when using a single thread, in 18s when using 6 threads but no parallel GC and in over a minute when using 6 threads with parallel GC! The true slowdown in the full code is actually worse and relevant for the application (some steps take >1 hour instead of <1 minute!). Parts of the code do take full advantage of parallel processing, this is just one simple test case -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11783> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11783: Very large slowdown when using parallel garbage collector -------------------------------------+------------------------------------- Reporter: luispedro | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 7.10.3 Resolution: | Keywords: performance, | garbage collector Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by luispedro): * Attachment "TestGC.hs" added. Source code for a simple test case -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11783> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11783: Very large slowdown when using parallel garbage collector -------------------------------------+------------------------------------- Reporter: luispedro | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 7.10.3 Resolution: | Keywords: performance, | garbage collector Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by luispedro): * Attachment "RunTimeGC.sh" added. Bash script to generate test data and run the code -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11783> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11783: Very large slowdown when using parallel garbage collector -------------------------------------+------------------------------------- Reporter: luispedro | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 7.10.3 Resolution: | Keywords: performance, | garbage collector Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Description changed by luispedro: @@ -11,1 +11,15 @@ - simple test case + simple test case. + + On some machines it seems worse than others and it seems that the input + file (data.txt) needs to be quite large for the problem to really show up + (the attached script generates a 16 million input file, this is still + smaller than some of my real use cases, but I couldn't trigger it with + only 1 million). Similarly, with 4 threads, the slowdown is detectable, + but not as large. + + While running, CPU usage is very high (I tested with 16 threads and it + uses 16 CPUs continuously, top reports 1600% CPU). + + Using '+RTS -A64m' is another way around the issue, but for the full + application it is still not as effective as '+RTS -qg', so there still + seems to be a performance issue here. New description: As part of debugging some performance issues on an application I am writing, I concluded that the issue is in the parallel GC implemented in the GHC RTS. I extracted the code attached to make a self-contained use- case, but in my system the code runs in 16s when using a single thread, in 18s when using 6 threads but no parallel GC and in over a minute when using 6 threads with parallel GC! The true slowdown in the full code is actually worse and relevant for the application (some steps take >1 hour instead of <1 minute!). Parts of the code do take full advantage of parallel processing, this is just one simple test case. On some machines it seems worse than others and it seems that the input file (data.txt) needs to be quite large for the problem to really show up (the attached script generates a 16 million input file, this is still smaller than some of my real use cases, but I couldn't trigger it with only 1 million). Similarly, with 4 threads, the slowdown is detectable, but not as large. While running, CPU usage is very high (I tested with 16 threads and it uses 16 CPUs continuously, top reports 1600% CPU). Using '+RTS -A64m' is another way around the issue, but for the full application it is still not as effective as '+RTS -qg', so there still seems to be a performance issue here. -- -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11783#comment:1> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11783: Very large slowdown when using parallel garbage collector -------------------------------------+------------------------------------- Reporter: luispedro | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 8.0.2 Component: Runtime System | Version: 7.10.3 Resolution: | Keywords: performance, | garbage collector Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by simonmar): * owner: => simonmar * priority: normal => high * milestone: => 8.0.2 Comment: I'm able to reproduce this, but I don't know what the cause is yet. Thanks for the test case! -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11783#comment:2> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11783: Very large slowdown when using parallel garbage collector -------------------------------------+------------------------------------- Reporter: luispedro | Owner: simonmar Type: bug | Status: new Priority: high | Milestone: 8.0.2 Component: Runtime System | Version: 7.10.3 Resolution: | Keywords: performance, | garbage collector Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by Simon Marlow <marlowsd@…>): In [changeset:"5c4cd0e44657d52f7ca5fee63f8765d17f1fbe85/ghc" 5c4cd0e4/ghc]: {{{ #!CommitTicketReference repository="ghc" revision="5c4cd0e44657d52f7ca5fee63f8765d17f1fbe85" Cache the size of part_list/scavd_list (#11783) After a parallel GC, it is possible to have a long list of blocks in ws->part_list, if we did a lot of work stealing but didn't fill up the blocks we stole. These blocks persist until the next load-balanced GC, which might be a long time, and during every GC we were traversing this list to find its size. The fix is to maintain the size all the time, so we don't have to compute it. }}} -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11783#comment:3> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11783: Very large slowdown when using parallel garbage collector -------------------------------------+------------------------------------- Reporter: luispedro | Owner: simonmar Type: bug | Status: merge Priority: high | Milestone: 8.0.2 Component: Runtime System | Version: 7.10.3 Resolution: | Keywords: performance, | garbage collector Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by simonmar): * status: new => merge -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11783#comment:4> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
#11783: Very large slowdown when using parallel garbage collector -------------------------------------+------------------------------------- Reporter: luispedro | Owner: simonmar Type: bug | Status: closed Priority: high | Milestone: 8.0.1 Component: Runtime System | Version: 7.10.3 Resolution: fixed | Keywords: performance, | garbage collector Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: merge => closed * resolution: => fixed * milestone: 8.0.2 => 8.0.1 Comment: Merged to `ghc-8.0` as 9649973c8038591a2c9f77e1183dd920b724daa5. -- Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11783#comment:5> GHC <http://www.haskell.org/ghc/> The Glasgow Haskell Compiler
participants (1)
-
GHC