#8732: Global big object heap allocator lock causes contention -------------------------------------+------------------------------------ Reporter: tibbe | Owner: simonmar Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: None/Unknown | Difficulty: Unknown Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Description changed by tibbe: Old description:

The lock allocate() takes when allocating big objects hurts scalability of I/O bound application. Network.Socket.ByteString.recv is typically called with a buffer size of 4096, which causes a ByteString of that size to be allocate. The size of this ByteString causes it to be allocated from the big object space, which causes contention of the global lock that guards that space.

See http://www.yesodweb.com/blog/2014/02/new-warp for a real world example.

New description: The lock `allocate` takes when allocating big objects hurts scalability of I/O bound application. `Network.Socket.ByteString.recv` is typically called with a buffer size of 4096, which causes a `ByteString` of that size to be allocated. The size of this `ByteString` causes it to be allocated from the big object space, which causes contention of the global lock that guards that space. See http://www.yesodweb.com/blog/2014/02/new-warp for a real world example. -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8732#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#8732: Global big object heap allocator lock causes contention --------------------------------------------+------------------------------ Reporter: tibbe | Owner: simonmar Type: bug | Status: new Priority: normal | Milestone: 7.10.1 Component: Runtime System | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Runtime performance bug | Unknown/Multiple Test Case: | Difficulty: Unknown Blocking: | Blocked By: | Related Tickets: --------------------------------------------+------------------------------ Comment (by ezyang): It is a good thing that these blocks are considered big blocks, since we don't really want to be copying the buffers around. So one thought might be to make the large block list in generation-0 per-thread, and perform allocations from a thread-local block list. But you have to be careful: objects that are larger than a block need contiguous blocks, so unless you are only going to enable this for large objects that still fit in a single block, you'll have to maintain multiple lists with the sizes you want. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8732#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#8732: Global big object heap allocator lock causes contention --------------------------------------------+------------------------------ Reporter: tibbe | Owner: simonmar Type: bug | Status: new Priority: normal | Milestone: 7.10.1 Component: Runtime System | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Runtime performance bug | Unknown/Multiple Test Case: | Difficulty: Unknown Blocking: | Blocked By: | Related Tickets: --------------------------------------------+------------------------------ Comment (by ezyang): It's pretty standard, yes (we implement for handling the global block pool), but it does mean all of that code would have to be made thread- local. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8732#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#8732: Global big object heap allocator lock causes contention --------------------------------------------+------------------------------ Reporter: tibbe | Owner: simonmar Type: bug | Status: new Priority: normal | Milestone: 7.10.1 Component: Runtime System | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Runtime performance bug | Unknown/Multiple Test Case: | Difficulty: Unknown Blocking: | Blocked By: | Related Tickets: --------------------------------------------+------------------------------ Comment (by carter): @tibbe, becaue TLS is slow on OS X currently? (mind you, my understanding is that the other RTS issues go away when building GHC with a REAL GCC, right? I take it thats not the case for this discussion? ) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8732#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#8732: Global big object heap allocator lock causes contention --------------------------------------------+------------------------------ Reporter: tibbe | Owner: simonmar Type: bug | Status: new Priority: normal | Milestone: 7.10.1 Component: Runtime System | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Runtime performance bug | Unknown/Multiple Test Case: | Difficulty: Unknown Blocking: | Blocked By: | Related Tickets: --------------------------------------------+------------------------------ Comment (by simonmar): I don't really understand why in mighty he couldn't just re-use the same block. I'm kind of surprised that this is a bottleneck, and I think it needs more investigation. We only take the lock for large objects, so typically there's going to be a lot of computation going on per allocation. I suppose if it really is a problem then we could just have a per-thread block pool at the granularity of a megablock to avoid fragmentation issues. We just push the global lock back to the megablock free list. This has the danger that we might have a lot of free blocks owned by one thread that don't get used, though, so we might want to redistribute the free blocks at GC. Things start to get annoyingly complicated. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8732#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#8732: Global big object heap allocator lock causes contention -------------------------------------+------------------------------------- Reporter: tibbe | Owner: simonmar Type: bug | Status: new Priority: normal | Milestone: 7.10.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: performance bug | Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Changes (by ihameed): * cc: idhameed@… (added) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8732#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#8732: Global big object heap allocator lock causes contention -------------------------------------+------------------------------------- Reporter: tibbe | Owner: simonmar Type: bug | Status: new Priority: normal | Milestone: 7.10.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Unknown Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: performance bug | Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Comment (by ezyang): carter: contiguous heap has not been merged in, and it doesn't really help for this problem. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8732#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler

#8732: Global big object heap allocator lock causes contention -------------------------------------+------------------------------------- Reporter: tibbe | Owner: simonmar Type: bug | Status: new Priority: normal | Milestone: 7.12.1 Component: Runtime System | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Runtime | Unknown/Multiple performance bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by simonmar): Malloc is fine for ByteStrings, but we can't use it for heap-resident objects due to the way block descriptors work. Our memory is always MB- aligned, so that we can put the block descriptors at the beginning of the MB. Also the GC has to be able to distinguish heap memory from non-heap memory, and we currently take advantage of the fact that memory is allocated in MB chunks to reduce the granularity that we have to map the address space. The contiguous-heap patch solves this in a different way (that is also incompatible with malloc). -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8732#comment:16 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler