[GHC] #14492: Tiered memory allocation restricts available memory
#14492: Tiered memory allocation restricts available memory -------------------------------------+------------------------------------- Reporter: unode | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime | Version: 8.0.2 System | Keywords: memory ulimit | Operating System: Linux Architecture: x86_64 | Type of failure: Poor/confusing (amd64) | error message Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- This was first noticed when running a binary produced with GHC 8.0.2 through a queueing system (Sun Grid Engine) that limits available memory by setting `ulimit -v`. When running with **unlimited** address space, the binary reserves 1TB of RAM and everything works as expected. When running with **limited** address space, reservation happens in powers of 2 leading to significant unused memory at higher memory values. The tests are summarized in the following table: {{{ Limit VIRT Unused (all in GB) 8 4 8 10 8 2 16 8 8 32 16 16 34 32 2 64 32 32 66 64 2 128 64 64 130 128 2 256 128 128 258 256 2 512 256 256 514 512 2 1024 512 512 }}} In the last line, setting `ulimit -v` to 1024GB (1073741824) causes the binary to only be able to reserve 512GB. If at runtime the software needs more than 512GB it will abort with "out of memory" instead of expanding to as close to the specified ulimit as possible. PS: Is there any way to make the "out of memory" error a little bit more informative? Something that informs the user of how much memory was already being used and how much more was going to be reserved when the failure was seen? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14492 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14492: Tiered memory allocation restricts available memory -------------------------------------+------------------------------------- Reporter: unode | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: memory ulimit Operating System: Linux | Architecture: x86_64 Type of failure: Poor/confusing | (amd64) error message | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari): I suppose we could set an upper bound on the reservation size. That is, exponentially increase up to, e.g., 32GB and then from there allocate 32 GB at a time. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14492#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14492: Tiered memory allocation restricts available memory -------------------------------------+------------------------------------- Reporter: unode | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: memory ulimit Operating System: Linux | Architecture: x86_64 Type of failure: Poor/confusing | (amd64) error message | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by bgamari):
PS: Is there any way to make the "out of memory" error a little bit more informative? Something that informs the user of how much memory was already being used and how much more was going to be reserved when the failure was seen?
Probably; unfortunately there are a number of places where this error is thrown scattered about the runtime. This should also be fixed. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14492#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14492: Tiered memory allocation restricts available memory -------------------------------------+------------------------------------- Reporter: unode | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: memory ulimit Operating System: Linux | Architecture: x86_64 Type of failure: Poor/confusing | (amd64) error message | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by unode): Replying to [comment:1 bgamari]:
I suppose we could set an upper bound on the reservation size. That is, exponentially increase up to, e.g., 32GB and then from there allocate 32 GB at a time.
32GB might still be quite large of a slice. On a 64GB machine that would be half of the total, making the software use at most ~32GB (since 64GB wouldn't be reservable). I don't know anything about GHC's internals so take this with a grain of salt. Would it make sense to instead half the previous reservation and keep halfing if unsuccessful? Otherwise, on systems that rely on this `ulimit -v` behavior, there will always be a gap between possible memory and what the software is actually able to use. My current understanding is that a process tries to reserve memory and doesn't try to change this value during runtime. Is it correct to say that any effort (or lack thereof) at start will constrain execution? If so shouldn't a "reserve as much as the system allows" effort be the default? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14492#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14492: Tiered memory allocation restricts available memory -------------------------------------+------------------------------------- Reporter: unode | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: memory ulimit Operating System: Linux | Architecture: x86_64 Type of failure: Poor/confusing | (amd64) error message | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by luispedro): The current behaviour is to start at 1TB and divide by 2 until success (or failure if it cannot allocate at least 1MB): https://github.com/ghc/ghc/blob/25f36bd7ba6899be6c720292528c56bc35e0f089/rts... If the memory available is in between two multiples of 2, then this can lead to a lot of unused memory. A (not so implausible, I think our systems might run something similar) worst-case scenario is running on a system with 1T of RAM, but where the sysadmin has wisely chosen to reserve 1GB for a monitoring process. Now, a whole 511GB of RAM are wasted! Btw, that code mentions https://ghc.haskell.org/trac/ghc/ticket/10877 with a previous discussion on this issue. A simple solution is to replace `*len /= 2` by `*len -= *len/8`. The max number of calls to mmap() goes up from 14 to 67, but wastage is now at most 1/8th of the final allocated memory. (Another question is whether a GHC compiled programme could access >1TB of RAM? We now have a few 4TB machines and more on the way...) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14492#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14492: Tiered memory allocation restricts available memory -------------------------------------+------------------------------------- Reporter: unode | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.4.1 Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: memory ulimit Operating System: Linux | Architecture: x86_64 Type of failure: Poor/confusing | (amd64) error message | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * cc: Jaffacake (added) * milestone: => 8.4.1 Comment:
A simple solution is to replace `*len /= 2` by `*len -= *len/8`. The max number of calls to `mmap()` goes up from 14 to 67, but wastage is now at most 1/8th of the final allocated memory.
Indeed this sounds like a pretty simple solution. I don't even think the increase in number of `mmap` calls is all that bad since these are failed `mmap`s, so they shouldn't incur the usual overhead. What do you think, simonmar? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14492#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14492: Tiered memory allocation restricts available memory -------------------------------------+------------------------------------- Reporter: unode | Owner: (none) Type: bug | Status: new Priority: normal | Milestone: 8.4.1 Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: memory ulimit Operating System: Linux | Architecture: x86_64 Type of failure: Poor/confusing | (amd64) error message | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4215 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * differential: => Phab:D4215 Comment: Here is a quick patch. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14492#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14492: Tiered memory allocation restricts available memory -------------------------------------+------------------------------------- Reporter: unode | Owner: (none) Type: bug | Status: patch Priority: normal | Milestone: 8.4.1 Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: memory ulimit Operating System: Linux | Architecture: x86_64 Type of failure: Poor/confusing | (amd64) error message | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4215 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: new => patch -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14492#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14492: Tiered memory allocation restricts available memory
-------------------------------------+-------------------------------------
Reporter: unode | Owner: (none)
Type: bug | Status: patch
Priority: normal | Milestone: 8.4.1
Component: Runtime System | Version: 8.0.2
Resolution: | Keywords: memory ulimit
Operating System: Linux | Architecture: x86_64
Type of failure: Poor/confusing | (amd64)
error message | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s): Phab:D4215
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by bgamari):
Another option, as suggested by Jaffacake on Diff:D4215, would be to probe
the limit. This test program
{{{#!c
#include
#14492: Tiered memory allocation restricts available memory -------------------------------------+------------------------------------- Reporter: unode | Owner: bgamari Type: bug | Status: patch Priority: normal | Milestone: 8.6.1 Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: memory ulimit Operating System: Linux | Architecture: x86_64 Type of failure: Poor/confusing | (amd64) error message | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4215 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * owner: (none) => bgamari Comment: I'll try to take care of this. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14492#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14492: Tiered memory allocation restricts available memory -------------------------------------+------------------------------------- Reporter: unode | Owner: bgamari Type: bug | Status: patch Priority: normal | Milestone: 8.6.1 Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: memory ulimit Operating System: Linux | Architecture: x86_64 Type of failure: Poor/confusing | (amd64) error message | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4215 Wiki Page: | -------------------------------------+------------------------------------- Comment (by luispedro): Just to ping everyone on this. We are still having to over allocate memory on our servers to work around this bug, while the simple 1-liner above (`*len -= *len/8`) would get us a 90% solution. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14492#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14492: Tiered memory allocation restricts available memory -------------------------------------+------------------------------------- Reporter: unode | Owner: bgamari Type: bug | Status: patch Priority: high | Milestone: 8.6.1 Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: memory ulimit Operating System: Linux | Architecture: x86_64 Type of failure: Poor/confusing | (amd64) error message | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4215 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * priority: normal => high Comment: Right, I have a patch pending that I'll try to dust off. Thanks for the ping! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14492#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14492: Tiered memory allocation restricts available memory
-------------------------------------+-------------------------------------
Reporter: unode | Owner: bgamari
Type: bug | Status: patch
Priority: high | Milestone: 8.6.1
Component: Runtime System | Version: 8.0.2
Resolution: | Keywords: memory ulimit
Operating System: Linux | Architecture: x86_64
Type of failure: Poor/confusing | (amd64)
error message | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s): Phab:D4215
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Ben Gamari
#14492: Tiered memory allocation restricts available memory -------------------------------------+------------------------------------- Reporter: unode | Owner: bgamari Type: bug | Status: patch Priority: high | Milestone: 8.6.1 Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: memory ulimit Operating System: Linux | Architecture: x86_64 Type of failure: Poor/confusing | (amd64) error message | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4215 Wiki Page: | -------------------------------------+------------------------------------- Comment (by unode): bgamari thanks for the fix. There's also the 1TB maximum reservation that luispedro mentioned on comment:4 which seems to be set on https://git.haskell.org/ghc.git/blob/730781b42a003604cfa047a02280757a07b0958... Implementing the probing solution you proposed on comment:8 would also address the 1TB limit.\\ Did you manage to get anywhere with that approach? Thanks again. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14492#comment:14 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14492: Tiered memory allocation restricts available memory -------------------------------------+------------------------------------- Reporter: unode | Owner: bgamari Type: bug | Status: patch Priority: high | Milestone: 8.6.1 Component: Runtime System | Version: 8.0.2 Resolution: | Keywords: memory ulimit Operating System: Linux | Architecture: x86_64 Type of failure: Poor/confusing | (amd64) error message | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4215 Wiki Page: | -------------------------------------+------------------------------------- Comment (by unode): PS: On your patch above if I use: {{{ #!diff - printf("%s: cur: %d\n", name, lim.rlim_cur); - printf("%s: max: %d\n", name, lim.rlim_max); + printf("%s: cur: %lu\n", name, lim.rlim_cur); + printf("%s: max: %lu\n", name, lim.rlim_max); }}} I get correct values from the test program, both for AS and RSS. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14492#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14492: Tiered memory allocation restricts available memory
-------------------------------------+-------------------------------------
Reporter: unode | Owner: bgamari
Type: bug | Status: patch
Priority: high | Milestone: 8.6.1
Component: Runtime System | Version: 8.0.2
Resolution: | Keywords: memory ulimit
Operating System: Linux | Architecture: x86_64
Type of failure: Poor/confusing | (amd64)
error message | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s): Phab:D4215
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Ben Gamari
#14492: Tiered memory allocation restricts available memory -------------------------------------+------------------------------------- Reporter: unode | Owner: bgamari Type: bug | Status: closed Priority: high | Milestone: 8.6.1 Component: Runtime System | Version: 8.0.2 Resolution: fixed | Keywords: memory ulimit Operating System: Linux | Architecture: x86_64 Type of failure: Poor/confusing | (amd64) error message | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Phab:D4215 Wiki Page: | -------------------------------------+------------------------------------- Changes (by bgamari): * status: patch => closed * resolution: => fixed -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/14492#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#14492: Tiered memory allocation restricts available memory
-------------------------------------+-------------------------------------
Reporter: unode | Owner: bgamari
Type: bug | Status: closed
Priority: high | Milestone: 8.6.1
Component: Runtime System | Version: 8.0.2
Resolution: fixed | Keywords: memory ulimit
Operating System: Linux | Architecture: x86_64
Type of failure: Poor/confusing | (amd64)
error message | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s): Phab:D4215
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Ben Gamari
participants (1)
-
GHC