[GHC] #8604: Some stack/vmem limits (ulimit) combinations causing GHC to fail
#8604: Some stack/vmem limits (ulimit) combinations causing GHC to fail ----------------------------------+---------------------------------- Reporter: clavin | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.6.3 Keywords: | Operating System: Other Architecture: x86_64 (amd64) | Type of failure: Runtime crash Difficulty: Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | ----------------------------------+---------------------------------- I have encountered a strange occurrence with GHC that was causing several GHC job failures on an SGE cluster. It turned out that there were other SGE users who needed an absurdly large stack size limit (set via 'ulimit -s') in the several gigabytes range. The default stack size limit had to be raised for the entire cluster. Any job that was run on a machine where the virtual memory limit was less than or equal to 2X the stack size limit, GHC would crash with the following message: ghc: failed to create OS thread: Cannot allocate memory I am running on GHC 7.6.3 with a 64-bit RedHat Enterprise OS, version 5.5. To reproduce the error, I was able to create the following test case: [ ~]$ ulimit -v unlimited [ ~]$ ulimit -s 10240 [ ~]$ ghc --version The Glorious Glasgow Haskell Compilation System, version 7.6.3 [ ~]$ ulimit -v 200000 [ ~]$ ulimit -s 100000 [ ~]$ ghc --version ghc: failed to create OS thread: Cannot allocate memory Several other programs work find using these settings, but GHC consistently has problems. Is this a fundamental issue with how GHC operates or can this be addressed? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8604 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8604: Some stack/vmem limits (ulimit) combinations causing GHC to fail ----------------------------------+---------------------------------- Reporter: clavin | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Other | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Difficulty: Unknown Test Case: | Blocked By: Blocking: | Related Tickets: ----------------------------------+---------------------------------- Comment (by carter): ghc allocates the stack on the heap, so this may be tripping a corner case somehow... could you try it out with ghc HEAD to see if the problem is still there? theres been some changes to the default way stacks work. (that said, definitely worth understanding) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8604#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
ghc allocates the stack on the heap, so this may be tripping a corner case somehow...
could you try it out with ghc HEAD to see if the problem is still there?
#8604: Some stack/vmem limits (ulimit) combinations causing GHC to fail ----------------------------------+---------------------------------- Reporter: clavin | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Other | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Difficulty: Unknown Test Case: | Blocked By: Blocking: | Related Tickets: ----------------------------------+---------------------------------- Comment (by clavin): Replying to [comment:1 carter]: theres been some changes to the default way stacks work. (that said, definitely worth understanding) I compiled 7.6.3 from source, what would be the easiest way for me to do the same with GHC HEAD (I'm new around here)? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8604#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8604: Some stack/vmem limits (ulimit) combinations causing GHC to fail ----------------------------------+---------------------------------- Reporter: clavin | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Other | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Difficulty: Unknown Test Case: | Blocked By: Blocking: | Related Tickets: ----------------------------------+---------------------------------- Comment (by carter): pretty much the exact same process. Make sure you have the newest versions of happy and alex installed, then follow the directions here https://ghc.haskell.org/trac/ghc/wiki/Building basically: clone the ghc repo ./sync-all get perl boot ./configure make -j # nb, this does an optimized build, so it take a while, edit the build to be a fast one, docs for that are in the build guide then test the ghc-stage2 in the ./inplace/bin folder -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8604#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8604: Some stack/vmem limits (ulimit) combinations causing GHC to fail ----------------------------------+---------------------------------- Reporter: clavin | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Other | Architecture: x86_64 (amd64) Type of failure: Runtime crash | Difficulty: Unknown Test Case: | Blocked By: Blocking: | Related Tickets: ----------------------------------+---------------------------------- Comment (by int-e): Replying to [comment:1 carter]:
ghc allocates the stack on the heap, so this may be tripping a corner case somehow...
There are two kinds of stacks. The STG stack (one per Haskell thread) is indeed allocated on the heap, grown dynamically, and its maximum size is governed by the '-K' RTS option. But each OS thread also has its own C stack and the size of that is determined by the stack size resource limit by default: ({{{createOSThread()}}} in {{{rts/posix/OSThreads.c}}} uses {{{pthread_create()}}} with no specific thread attributes. Quoting {{{man pthread_create}}}: {{{ On Linux/x86-32, the default stack size for a new thread is 2 megabytes. Under the NPTL threading implementation, if the RLIMIT_STACK soft resource limit at the time the program started has any value other than "unlimited", then it determines the default stack size of new threads. }}} Other implementations may differ. I'm not sure whether or how this should be dealt with by ghc. In some circumstances, a large C stack is required (depending on used foreign libraries), and currently the soft stack resource limit is the knob to turn in that case. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8604#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8604: Some stack/vmem limits (ulimit) combinations causing GHC to fail -------------------------------------+------------------------------------- Reporter: clavin | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: | Version: 7.6.3 Documentation | Keywords: Resolution: | Architecture: x86_64 (amd64) Operating System: Other | Difficulty: Unknown Type of failure: | Blocked By: Documentation bug | Related Tickets: Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Changes (by thomie): * failure: Runtime crash => Documentation bug * component: Compiler => Documentation Comment: Maybe part of int-e's comment:4 could go into to [https://www.haskell.org/ghc/docs/7.8.3/html/users_guide/runtime- control.html#setting-rts-options user's guide]. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8604#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8604: Some stack/vmem limits (ulimit) combinations causing GHC to fail -------------------------------------+------------------------------------- Reporter: clavin | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: | Version: 7.6.3 Documentation | Keywords: Resolution: | Architecture: x86_64 (amd64) Operating System: Other | Difficulty: Unknown Type of failure: | Blocked By: Documentation bug | Related Tickets: Test Case: | Blocking: | Differential Revisions: | -------------------------------------+------------------------------------- Description changed by thomie: Old description:
I have encountered a strange occurrence with GHC that was causing several GHC job failures on an SGE cluster. It turned out that there were other SGE users who needed an absurdly large stack size limit (set via 'ulimit -s') in the several gigabytes range. The default stack size limit had to be raised for the entire cluster.
Any job that was run on a machine where the virtual memory limit was less than or equal to 2X the stack size limit, GHC would crash with the following message:
ghc: failed to create OS thread: Cannot allocate memory
I am running on GHC 7.6.3 with a 64-bit RedHat Enterprise OS, version 5.5.
To reproduce the error, I was able to create the following test case:
[ ~]$ ulimit -v unlimited [ ~]$ ulimit -s 10240 [ ~]$ ghc --version The Glorious Glasgow Haskell Compilation System, version 7.6.3 [ ~]$ ulimit -v 200000 [ ~]$ ulimit -s 100000 [ ~]$ ghc --version ghc: failed to create OS thread: Cannot allocate memory
Several other programs work find using these settings, but GHC consistently has problems. Is this a fundamental issue with how GHC operates or can this be addressed?
New description: I have encountered a strange occurrence with GHC that was causing several GHC job failures on an SGE cluster. It turned out that there were other SGE users who needed an absurdly large stack size limit (set via 'ulimit -s') in the several gigabytes range. The default stack size limit had to be raised for the entire cluster. Any job that was run on a machine where the virtual memory limit was less than or equal to 2X the stack size limit, GHC would crash with the following message: ghc: failed to create OS thread: Cannot allocate memory I am running on GHC 7.6.3 with a 64-bit RedHat Enterprise OS, version 5.5. To reproduce the error, I was able to create the following test case: [ ~]$ ulimit -v unlimited [ ~]$ ulimit -s 10240 [ ~]$ ghc --version The Glorious Glasgow Haskell Compilation System, version 7.8.3 [ ~]$ ulimit -v 2000000 [ ~]$ ulimit -s 1000000 [ ~]$ ghc --version ghc: failed to create OS thread: Cannot allocate memory [ ~]$ ulimit -s 500000 [ ~]$ ghc --version The Glorious Glasgow Haskell Compilation System, version 7.8.3 Several other programs work find using these settings, but GHC consistently has problems. Is this a fundamental issue with how GHC operates or can this be addressed? -- -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8604#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8604: Some stack/vmem limits (ulimit) combinations causing GHC to fail -------------------------------------+------------------------------------- Reporter: clavin | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Documentation | Version: 7.6.3 Resolution: | Keywords: Operating System: Other | Architecture: x86_64 Type of failure: Documentation | (amd64) bug | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by rwbarton): I'm wary of trying to document the underlying behavior of the operating system in the user's guide, especially when that behavior is likely to vary between different operating systems. It seems that the behavior of ghc (or any program built with the threaded runtime) is no different than any other program that starts multiple OS threads. Granted it might not be obvious to the user that ghc itself starts multiple threads, but then they will never know where to look in the documentation to learn this. But I guess we could at least clarify in the documentation for some or all of the `-K/-k*` options that they refer to the Haskell stack, not the C stack. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8604#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC