[GHC] #12492: internal error: allocation of 1048608 bytes too large
#12492: internal error: allocation of 1048608 bytes too large -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Runtime | Version: 8.0.1 System | Keywords: | Operating System: Unknown/Multiple Architecture: | Type of failure: None/Unknown Unknown/Multiple | Test Case: | Blocked By: Blocking: | Related Tickets: Differential Rev(s): | Wiki Page: -------------------------------------+------------------------------------- I'm getting this error error when trying to use the `ghc-datasize` package from Hackage to calculate the in memory size of a data structure. The full error is: {{{ load-test: internal error: allocation of 1048608 bytes too large (GHC should have complained at compile-time) (GHC version 8.0.1 for x86_64_unknown_linux) Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug }}} However, this message is bogus because this is not a static data structure known at compile time, but data loaded from disk. Currently, my test program just loads this complex data structure (about 27 megabytes of CSV with embedded JSON on disk) into memory and then calls `recursiveSize` from `GHC.Datasize` on it. I get this same error message with both ghc7.10.3 and ghc-8.0.1. If I remove the call to `recursiveSize` there is no error. If I `Control.Deepseq.force` the data structures before calling `recursiveSize` there is no error. So, who is to blame? `ghc-datasize` is just Haskell code that calls public APIs so I think the blame lies with GHC. I'll work on getting a small re-producable test case. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12492 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#12492: internal error: allocation of 1048608 bytes too large -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Runtime System | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonmar): @erikd the message indicates that somehow the compiled program attempted to allocate 1048608 in one go, which is supposed to cause a compile-time error. It is not related to whether the data being processed is static or dynamic, but has to do with how the compiled code allocates memory. I don't know what's causing it, but it shouldn't be too hard to track down - find out which bit of code is causing the offending allocation, and find out why GHC didn't detect the overlarge allocation at compile-time. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12492#comment:1 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#12492: internal error: allocation of 1048608 bytes too large -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Runtime System | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by erikd): I suspect this issue was actually a problem with an attoparsec parser not handing escape sequences correctly (my fault) and then causing an allocation that was too big. Still not understanding how this could have been detected at compile time. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12492#comment:2 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#12492: internal error: allocation of 1048608 bytes too large -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Runtime System | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by erikd): The parser bug has been fixed and all the Attoparsec parsers now check that all input has been consumed. I thought that this had killed this problem, but then it reared its ugly head again. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12492#comment:3 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#12492: internal error: allocation of 1048608 bytes too large -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Runtime System | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by erikd): Running with the debug RTS and `-Ds` I get: {{{ cap 0: finished GC task exiting cap 0: running thread 1 (ThreadRunGHC) cap 0: thread 1 stopped (yielding) cap 0: running thread 1 (ThreadRunGHC) cap 0: thread 1 stopped (heap overflow) load-test: internal error: allocation of 1034912 bytes too large (GHC should have complained at compile-time) (GHC version 7.10.3 for x86_64_unknown_linux) Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug }}} To me, "heap overflow" sounds bad. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12492#comment:4 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#12492: internal error: allocation of 1048608 bytes too large -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Runtime System | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by erikd): Looking at function `scheduleHandleHeapOverflow` in `rts/Schedule.c` it looks like its not possible to allocate a single block larger than `252 * 4096` bytes. Is that right? I *know* I've allocated `ByteString`s bigger than that or are `ByteString`s allocated in a different way? -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12492#comment:5 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#12492: internal error: allocation of 1048608 bytes too large -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Runtime System | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by erikd): Begining to think this is actually a problem with `ghc-datasize` or `ghc- heap-view` which it sits on top of. I have a function `readComplexDataStructureFromDisk`. If I do: {{{#!hs x <- readComplexDataStructureFromDisk size <- recursiveSize $ Control.DeepSeq.force x print size }}} I get the error above. If I rewrite it as: {{{#!hs x <- Control.DeepSeq.force <$> readComplexDataStructureFromDisk size <- recursiveSize x print size }}} there is no error. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12492#comment:6 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#12492: internal error: allocation of 1048608 bytes too large -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Runtime System | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by erikd): The `ghc-datasize` package I'm using depends on `ghc-heap-view` and @nomeata suggests that its a little brittle and possibly the problem here: https://github.com/nomeata/ghc-heap-view/issues/10 -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12492#comment:7 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#12492: internal error: allocation of 1048608 bytes too large -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Runtime System | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonmar): Regardless of what's going on in `ghc-datasize` and `ghc-heap-view`, if the GHC-compiled program emits this message then it's a bug in GHC, because we're supposed to fail at compile-time instead. (the number in the error message is a compile-time constant, not a runtime value) -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12492#comment:8 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#12492: internal error: allocation of 1048608 bytes too large -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Runtime System | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by rwbarton): Simon, is that true even in view of the fact that `ghc-heap-view` contains Cmm that calls `ALLOC_PRIM_P` directly? (https://github.com/nomeata/ghc- heap-view/blob/master/cbits/HeapViewPrim.cmm#L27) It's not clear to me how custom Cmm is supposed to allocate heap objects that might be either small or large, with small objects allocated into the nursery for efficiency. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12492#comment:9 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#12492: internal error: allocation of 1048608 bytes too large -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: new Priority: normal | Milestone: 8.0.2 Component: Runtime System | Version: 8.0.1 Resolution: | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Comment (by simonmar): Ah, I didn't realise it had custom Cmm. Yes, that could account for the observed behaviour. I think it should be possible to use a conditional on the size of the object and call `allocate()` for a large object (with a `MAYBE_GC` check), or `ALLOC_PRIM_P` for a small object. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12492#comment:10 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
#12492: internal error: allocation of 1048608 bytes too large -------------------------------------+------------------------------------- Reporter: erikd | Owner: Type: bug | Status: closed Priority: normal | Milestone: 8.0.2 Component: Runtime System | Version: 8.0.1 Resolution: invalid | Keywords: Operating System: Unknown/Multiple | Architecture: | Unknown/Multiple Type of failure: None/Unknown | Test Case: Blocked By: | Blocking: Related Tickets: | Differential Rev(s): Wiki Page: | -------------------------------------+------------------------------------- Changes (by nomeata): * status: new => closed * resolution: => invalid Comment: I believe this can be closed; and further discussion should happen at https://github.com/nomeata/ghc-heap-view/issues/10, as this is clearly a problem with the hack that is called `ghc-heap-view`. -- Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/12492#comment:11 GHC http://www.haskell.org/ghc/ The Glasgow Haskell Compiler
participants (1)
-
GHC