[GHC] #8189: Default to infinite stack size?
#8189: Default to infinite stack size? ------------------------------------+------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.6.3 Keywords: | Operating System: Unknown/Multiple Architecture: Unknown/Multiple | Type of failure: Runtime crash Difficulty: Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | ------------------------------------+------------------------------------- http://www.haskell.org/pipermail/haskell-cafe/2013-August/108570.html It looks like any code of the form {{{ ... = do x1 <- m1 x2 <- m2 [do something with x1 and m2] }}} will stack overflow when used with a strict monad (like IO) and sufficiently large data to deal with (a 1m element list is enough to exceed the default 8MB stack size limit). Other examples include: {{{ sequence (m:ms) = do x <- m xs <- sequence ms return (x:xs) }}} and {{{ liftM2, liftM3, ... }}} By slightly modifying your program to do the same thing via heap allocation (e.g. using a difference list in sequence), a stack-overflow rejected program turns into a valid program. This means that valid and idiomatic code like {{{ xs <- replicateM n someIOAction xs <- mapM someIOAction [1..n] }}} will eventually crash, which has an impact on practically all Haskell software and libraries out there. For examples, see https://github.com/ndmitchell/shake/blob/e0e0a43/Development/Shake/Database.... https://github.com/haskell-suite/haskell-src-exts/issues/27 as well as the mapM usages in cabal, haddock, any database binding that allows querying a lists of results, etc. While some argue that ''"you shouldn't use lists of that size"'', ''"allocating a 150 MB list is wrong"'' or that they ''"carefully avoid any use of sequence/mapM due to the problem"'' pointed out above, I think that collecting a million results of a strict monadic action into a linked list using standard functions like mapM is a common pattern and valid use case. The stack based evaluation that GHC performs is an optimization and should be transparent to the user; it should not re-define some data to live in a fixed-size memory region, and it should work just as well as it does in GHCi. It seems that a default infinite stack size would mitigate this. A drawback might be that the stack space overflows we get so far to quickly spot a "wrong" program would not indicate certain errors any more. I currently doubt the usefulness of this debugging method, since it does not work reliably in the presence of optimization, doing the same thing via heap allocation makes the programs "correct", and it rejects valid programs if the input data is large enough. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------------+------------------------------------ Reporter: nh2 | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: Runtime crash | Difficulty: Unknown Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by nh2): Another way to phrase it is: If you have a language that is supposed to perform recursion on arbitrarily large input, and recursion is implemented with a stack, then the stack cannot be limited. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------------+------------------------------------ Reporter: nh2 | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: Runtime crash | Difficulty: Unknown Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by simonpj): I think that's a very reasonable position. It seems odd that you can have a very big list with no problem but not a very big stack. The only point I'd make is that if we make the stack infinite by default, and there's a stack overflow, the error message should say "Stack overflow on a stack of size 300Mbytes", not simply "Heap exhausted". Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------------+------------------------------------ Reporter: nh2 | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Compiler | Version: 7.6.3 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: Unknown/Multiple Type of failure: Runtime crash | Difficulty: Unknown Test Case: | Blocked By: Blocking: | Related Tickets: -------------------------------------+------------------------------------ Comment (by nh2): Simon,
The only point I'd make is that if we make the stack infinite by default, and there's a stack overflow, the error message should say "Stack overflow on a stack of size 300Mbytes", not simply "Heap exhausted".
I don't think I quite understand. If the stack size is unlimited, how can you get a stack overflow? Do you mean when all the memory is actually exhausted? Also in which cases would we simply print "Heap exhausted"? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Changes (by ezyang): * difficulty: Unknown => Easy (less than 1 hour) * component: Compiler => Runtime System Comment: Yes, you will still eventually run out of memory and get a heap overflow error. See [[GhcFile(rts/Schedule.c)]] for how we deal with this case; we actually service stack overflow errors during the normal operation of programs, because we use stack chunks and we need to allocate a new stack chunk when we run out. So actually, it's pretty simple to make sure we give the right error when we run out of heap, change this line in [[GhcFile(rts/Threads.c)]]: {{{ new_stack = (StgStack*) allocate(cap, chunk_size); }}} (as well as allocate) so that if requested heap allocation fails, it reports a stack overflow, not heap overflow. This should be pretty easy to do, patches welcome! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by nh2): The stack limit is not even reliable: http://www.reddit.com/r/haskell/comments/1luan1/strange_io_sequence_behaviou... -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by bgamari): I'm working on this in my `infinite-stack` branch https://github.com/bgamari/ghc/tree/infinite-stack. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by bgamari): I believe I'm pretty much done. The only thing I lack is the ability to report the correct stack size after allocation has failed. I believe the stack overflow message is produced by a hook in [[GhcFile(rts/hooks/StackOverflow.c)]] which is called by `stackOverflow` in [[GhcFile(rts/RtsUtils.c)]]. I'm having great difficulty tracing flow of control back further than this however. How does execution get from the `stackOverflow_closure` being thrown to `stackOverflow`? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by bgamari): I've finally wrapped up the remaining loose ends. This required changes to the RTS in addition to a minor change in `base` to account for an interface change in `stackOverflow` (`rts/RtsUtils.c`). These can be found at, * https://github.com/bgamari/ghc/compare/infinite-stack * https://github.com/bgamari/packages-base/compare/infinite-stack How should we proceed from here? Seeing as all of the changes are fairly straightforward, perhaps this could go in for 7.8? I'd be happy to write up something for the user manual. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size?
-------------------------------+-------------------------------------------
Reporter: nh2 | Owner:
Type: bug | Status: new
Priority: normal | Milestone:
Component: Runtime | Version: 7.6.3
System | Keywords:
Resolution: | Architecture: Unknown/Multiple
Operating System: | Difficulty: Easy (less than 1 hour)
Unknown/Multiple | Blocked By:
Type of failure: Runtime | Related Tickets:
crash |
Test Case: |
Blocking: |
-------------------------------+-------------------------------------------
Comment (by Austin Seipp
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by thoughtpolice): Merged in `d85044f6b201eae0a9e453b89c0433608e0778f0` with some minor tweaks. Notably, when `allocateFail(...) == NULL` when allocating `new_stack`, we call `stackOverflow(tso)` and then `stg_exit`, instead of `throwToSingleThreaded`. This is more in line with what `allocate` will now do. Thanks to Ben and Reid Barton for helping with this and spotting that error! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by ezyang): I'm a little late to the review party, but can we change the interface of allocateFail so that it returns only result type, and the allocated pointer is returned via an argument; i.e. use it like: {{{ allocateFail(Capability, W_, StgPtr*); StgPtr p; if (allocateFail(cap, n, &p)) { // allocation failed } else { // allocation ok; p points to memory } }}} -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by simonmar): There are a few things I'm not completely happy with here. By all means turn off the default stack limit, but the business with `allocateFail()` is misguided: `allocateFail()` will not return NULL. It only returns NULL when there is a single request for memory greater than the maximum heap size, and (a) by default there's no maximum heap size, and (b) even if there was, it would be highly unlikely to be smaller than the stack chunk size (32K). Did anyone test this? The `heapOverflow()` test in `allocate()` is purely optional, see #1791. The typical behaviour when someone writes a program that accidentally blows the stack will be for the machine to grind to a halt swapping, until the OS finally kills something (hopefully the right process). We have this behaviour for space leaks now, so I guess it's no worse to have it for stack leaks too. An actual "out of memory" error is rare, typically it happens when you try to allocate an array larger than the memory size, or something like that. Austin: could you back this out please, and let's discuss it some more. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:12 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by ezyang): Oops, sorry for not noticing that when I suggested allocate behaved that way. My bad. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:13 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size?
-------------------------------+-------------------------------------------
Reporter: nh2 | Owner:
Type: bug | Status: new
Priority: normal | Milestone:
Component: Runtime | Version: 7.6.3
System | Keywords:
Resolution: | Architecture: Unknown/Multiple
Operating System: | Difficulty: Easy (less than 1 hour)
Unknown/Multiple | Blocked By:
Type of failure: Runtime | Related Tickets:
crash |
Test Case: |
Blocking: |
-------------------------------+-------------------------------------------
Comment (by Austin Seipp
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: normal | Milestone: Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by thoughtpolice): Ah yes, I missed this behavior too. Sorry about that. I managed to trigger this by artificially limiting the heap and stack chunk size to be almost identical (or even `chunk_size` > heap size,) so early on an allocation for it would fail. I've reverted the relevant changes for the time being in `base` and `ghc`. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:15 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: high | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Changes (by thoughtpolice): * priority: normal => high * milestone: => 7.8.1 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:16 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: high | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by bgamari): Oh dear; that is embarrassing. My intent here was more along the lines of, `if (total_allocated > RtsFlags.GcFlags.maxHeapSize)`. While it seems that the total allocations of each capability individually is pretty easy to come by (`cap->total_allocated`), it's not clear how to get the sum of these without iterating over all capabilities which seems a bit excessive in a warm allocation path. Either way, comparing against `req_blocks` was certainly a bit sloppy. Regarding the lack of default heap size limit, perhaps a solution here might be to set the default heap limit to some fraction (say 0.8) of the physical memory size. Admittedly determining this might be a bit platform specific but seeing as the RTS already knows how to get a core count it's not a great stretch. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:17 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: high | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by simonmar): We can't tell whether the heap is full by looking at `total_allocated`, because the GC needs to run first (`total_allocated` basically tells you how much memory was allocated since the last GC). I can't think of a good way to determine whether we ran out of heap because a lot of stack was allocated - it's guesswork. Should there be a default maximum heap size? I'm hesitant to do this because enabling the maximum heap size also enables the automagic machinery that switches on the compacting collector when we get near the maximum heap size, and the compacting collector is much less well battle- tested than the standard copying GC, not to mention much slower. On the other hand having a maximum heap size with the copying GC doesn't work terribly well, because the copying GC wastes so much memory. We could just set the default maximum stack size to be some sensible fraction of real memory? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:18 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: high | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by bgamari): I see the issue with `total_allocations`. If you are hesitant to set a default heap limit I certainly won't disagree. It's not worth exposing potential bugs just to disable the stack limit. A larger default stack limit, on the other hand, is quite reasonable and accomplishes nearly all of what I initially set out to do. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:19 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: high | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by nh2): Replying to [comment:18 simonmar]:
We could just set the default maximum stack size to be some sensible fraction of real memory?
As sane as this sounds, it doesn't go well with unix philosophies (memory is to be treated as unlimited until allocation fails and memory/swap should be transparent to the user) and in practice (VMs need not report the actually available memory as physical memory, fast swap becomes useless). I think having this for the stack though is still much better than a default heap limit or the minimal heap limit we have had so far. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:20 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: new Priority: high | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by bgamari): I have another branch up at https://github.com/bgamari/ghc/compare/bump- stack-limit. In this branch I, * Introduce `getPhysicalMemorySize` to `rts/posix/` and `rts/win32`. Only the posix variant has been tested. * Fix `RtsUtils.c:stackOverflow` to print the actual stack size instead of the stack size limit. This requires a patch in `base` https://github.com/bgamari/packages-base/compare/infinite-stack * Allow RTS to take `-K 0` to disable the stack size limit. This is a hold-over from the last set and should arguably be dropped. * Set the default stack size limit to 80% of physical memory * Update the user guide appropriately Let me know how this looks. It would be great to get this in for 7.8. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:21 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: patch Priority: high | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Changes (by thoughtpolice): * status: new => patch -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:22 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: patch Priority: high | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by rwbarton): There are some issues of units here. The win32 `getPhysicalMemorySize` returns the physical memory size in bytes, while the posix one uses units of pages (need to multiply by `getPageSize()`), and `RtsFlags.GcFlags.maxStkSize` expects units of words (`sizeof(W_)`). Your `getPhysicalMemorySize` functions have the same unused variable as in #8289. A 32-bit system can easily have 4 GB (or even 16 GB) of physical memory, in which case the physical memory size in bytes (respectively, words) will overflow a `W_`. In this case, the stack size should be unlimited. I'm not sure if this is what the `if (maxStkSize <= 0)` test is supposed to be checking, but it won't work in general (imagine you somehow have just over 4 GB of RAM: then `getPhysicalMemorySize()` will overflow to a very small number). You should return `StgWord64` from `getPhysicalMemorySize` and use `StgWord64` when computing the default stack size, that should last a while. :) (FWIW I'm not totally sold on the idea of trying to guess an appropriate maximum stack size, but the alternatives of the current small stack limit or no stack limit at all aren't entirely satisfying either.) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:23 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: patch Priority: high | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by bgamari): Thanks for the review and good catch regarding the unit issue. I suppose this is what I get for trying to finish a patch on the bus. I've pushed a new set of patches. I was regarding #8289 as a separate issue and was waiting for a fix outside the tree but it's now fixed in this set. I agree that the arbitrary 80% stack limit isn't great but also fail to see any better options. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:24 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: patch Priority: high | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by simonmar): I added some comments on github. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:25 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: Type: bug | Status: patch Priority: high | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by bgamari): Replying to [comment:25 simonmar]:
I added some comments on github.
I believe I've now addressed all of these. It's still not entirely clear to me whether there is a consensus that breaking the interface with base (as done in https://github.com/bgamari/ghc/commit/871b193cc0c3450e6ce92ae4a0f3d90c5fc38c... and https://github.com/bgamari/packages- base/commit/78fb57c5834844b8bbc61833c7f0fac54e11bf36) is the right thing to do at this juncture. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:26 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: bgamari Type: bug | Status: patch Priority: high | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Changes (by bgamari): * owner: => bgamari Comment: It turns out this fix uncovered a bug in the testsuite which is fixed by https://github.com/bgamari/testsuite/commit/75bef4ae7c85cb73d0458ab6392a48c3.... -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:27 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
It's still not entirely clear to me whether there is a consensus that breaking the interface with base (as done in https://github.com/bgamari/ghc/commit/871b193cc0c3450e6ce92ae4a0f3d90c5fc38c... and https://github.com/bgamari/packages-
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: bgamari Type: bug | Status: patch Priority: high | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by simonmar): Replying to [comment:26 bgamari]: base/commit/78fb57c5834844b8bbc61833c7f0fac54e11bf36) is the right thing to do at this juncture. Looks fine to me. The RTS and base are tightly coupled. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:28 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: bgamari Type: bug | Status: patch Priority: high | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Changes (by bgamari): * cc: simonmar (added) Comment: What's the plan for merging this? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:29 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: bgamari Type: bug | Status: patch Priority: highest | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Changes (by simonpj): * priority: high => highest Comment: Since Simon M has approved, the milestone is 7.8.1 and the status is 'patch', I guess it's on Austin's patch queue. I'll up the priority since it seems like it should go in. Question: is there any user documentation? Should there be? Simon -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:30 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: bgamari Type: bug | Status: patch Priority: highest | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by bgamari): I updated the user manual to reflect the new `-K` default. That being said, the `bump-stack-limit` branch also includes a patch to optionally disable the stack size limit altogether. This hasn't been documented but should be. I'll take care of this ASAP. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:31 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: bgamari Type: bug | Status: patch Priority: highest | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Comment (by bgamari): A note describing how to disable the stack size limit has been added. I also rebased the branch against `master` to resolve a conflict. A new validation is running now. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:32 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------+------------------------------------------- Reporter: nh2 | Owner: bgamari Type: bug | Status: closed Priority: highest | Milestone: 7.8.1 Component: Runtime | Version: 7.6.3 System | Keywords: Resolution: fixed | Architecture: Unknown/Multiple Operating System: | Difficulty: Easy (less than 1 hour) Unknown/Multiple | Blocked By: Type of failure: Runtime | Related Tickets: crash | Test Case: | Blocking: | -------------------------------+------------------------------------------- Changes (by thoughtpolice): * status: patch => closed * resolution: => fixed Comment: This is now merged. Thanks Ben! -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:33 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------------+------------------------------------- Reporter: nh2 | Owner: bgamari Type: bug | Status: closed Priority: highest | Milestone: 7.8.1 Component: Runtime System | Version: 7.6.3 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Runtime crash | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by ekmett): If having an unbounded stack size was the default behavior the last reason that `mapM` has to live would go away. This would let us dramatically simplify `Traversable`. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:34 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#8189: Default to infinite stack size? -------------------------------------+------------------------------------- Reporter: nh2 | Owner: bgamari Type: bug | Status: closed Priority: highest | Milestone: 7.8.1 Component: Runtime System | Version: 7.6.3 Resolution: fixed | Keywords: Operating System: Unknown/Multiple | Architecture: Type of failure: Runtime crash | Unknown/Multiple Blocked By: | Test Case: Related Tickets: | Blocking: | Differential Revisions: -------------------------------------+------------------------------------- Comment (by carter): @ekmett, unbounded stack size (up to whatever limits the heap provides, or maybe 80% of the heap limit) IS the default now as of 7.8 onwards as far as I know. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/8189#comment:35 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC