[GHC] #9647: allocation of 10790760 bytes too large

29 Sep 2014

      #9647: allocation of 10790760 bytes too large
-------------------------------------+-------------------------------------
       Reporter:  mirko              |                   Owner:
           Type:  bug                |                  Status:  new
       Priority:  normal             |               Milestone:
      Component:  Compiler           |                 Version:  7.8.3
       Keywords:  run time           |        Operating System:  Windows
  insufficient memory                |         Type of failure:  Incorrect
   Architecture:  x86_64 (amd64)     |  warning at compile-time
     Difficulty:  Unknown            |               Test Case:
     Blocked By:                     |                Blocking:
Related Tickets:                     |  Differential Revisions:
-------------------------------------+-------------------------------------
 I got an error message and invitation to submit a bug
 report. So here is the error message first, followed
 by my description of what happened.

 C:\Users\Owner\Documents\hs>ghc -o void void.hs -rtsopts
 [1 of 1] Compiling Main             ( void.hs, void.o )
 Linking void.exe ...

 C:\Users\Owner\Documents\hs>void +RTS -K400M -RTS > void.txt
 void: internal error: allocation of 10790760 bytes too large
 (GHC should have complained at compile-time)
     (GHC version 7.8.3 for x86_64_unknown_mingw32)
     Please report this as a GHC bug:
 http://www.haskell.org/ghc/reportabug

 C:\Users\Owner\Documents\hs>void +RTS -K40000M -RTS > void.txt
 void: internal error: allocation of 10790760 bytes too large
 (GHC should have complained at compile-time)
     (GHC version 7.8.3 for x86_64_unknown_mingw32)
     Please report this as a GHC bug:
 http://www.haskell.org/ghc/reportabug

 Description of what happened.
 I gave my program really large numbers.
 It contains the following line:

 rr = (of40)*(r250^3041)

 In the above of40 is (if I remember correctly) the product
 of the first 40000 odd primes. I computed it with reduce
 (a computer algebra system) which saved the result (as an
 explicitly written out number in decimal notation), which
 I copied and inserted in my hs file, initializing of40 with
 its value. In the file it takes 2457 rows, 1024 cols, so
 it appears to be about 2 515 968 (that is two million+)
 decimal digits. As seen it is multiplied by r250^3041,
 here r250 is a 250-digit number so r250^3041 is about
 760 250 (seven hundred sixty thousand+) decimal digits,
 so the product rr = (of40)*(r250^3041) seems to be more
 than 3 000 000 (three million+) decimal digits.

 I tried to compile it yesterday, I did get an error
 message from the compiler: I do not remember exactly
 what it was, I may try to reproduce it later and include
 it, but it was something along the lines of insufficient
 amount of memory. I run a Hewlett-Packard HPE 500y,
 upgraded to 16 Gb RAM (the max it would take), Windows 7,
 AMD Phenom(tm) II X6 1045T Processor 2.70 GHz (6 core).
 So yesterday after I got the first error message I increased
 the paging file size from automatic to 32768 Mb initial, and
 65536 Mb max. Then tried to compile again, the compiler
 gradually took more and more memory (I was watching the performance
 using the Task Manager) so it gradually went up to using
 all 16 Gb of RAM, and stayed there for I think at least 40 min,
 could have been more than an hour (or even 2 hours, sorry
 can't quite remember). At some point it seemed the computer
 froze, the task manager stopped showing any activity, but I
 did not attempt to immediately restart it, the mouse would not
 respond anyway, and in a few minutes it fixed itself, it
 seemed to continue compiling, after which it eventually
 gave up and complained about insufficient amount
 of memory again. So this morning I increased
 the paging file to initial size 49152 Mb,
 max size 196608 Mb (I don't know if such big numbers
 make sense but I tried). I tried to compile again
 starting around 9am, left my apt at around 4pm and
 the compiler had not yet finished, and came back after
 midnight, the compiler had finished. It had produced
 a big exe file (I see now, around 9 pm, so it must
 have kept compiling for around 12 hours):
 09/27/2014  09:38 PM        20,602,960 void.exe
 AVG (antivirus) claimed that this file was a threat and
 refused to let me run it, so I had to disable AVG,
 after which I got the error messages as already
 enclosed at the beginning of this description.
 The Paging file reports as: Recommended 24574 Mb,
 Currently allocated: 163765 Mb.

 Note I gave the options -rtsopts to the compiler,
 I do not remember anymore what they mean, and whether
 they are valid (or perhaps not used anymore), I got
 them from a webpage perhaps a couple of years ago
 and at that time they made sense, so I just kept using them.
 I also gave the option +RTS -K400M -RTS when I tried
 to execute the compiled program, and then
 +RTS -K40000M -RTS, both versions generated the error
 messages shown above. I tend to believe that giving
 Haskell such large numbers inside the .hs file explains
 all of the above problems, it might even be a bit
 surprising it went so far before it finally gave up.
 But I do not really know, so let me just report it.

 So I changed the paging file back to automatic, and
 I am compiling the same file (after renaming it) to
 try to reproduce the error message that I got yesterday
 (or rather the other day, now being after midnight).
 It has been compiling for about 4 min so far, nothing
 really visible, memory usage stays at 1.28 Gb (where
 it was before I started compiling). I don't quite remember
 what happened the first time when I tried to compile,
 I think I got the error message (about insufficient
 amount of memory) immediately, or almost immediately.
 This does not happen now, I wonder if the computer
 took into account that I changed the paging file
 back to automatic, I may need to restart to make sure.
 Well, ok, 13 min after it started compiling, the memory
 usage started going up gradually. It goes from e.g.
 13.9 to 14 Gb in about a second or two. So in about
 2 min it went from 1.28 to 15.9 Gb, and now stays there
 (I do not know how to interpret all that Task Manager shows me).
 I may not be able to reproduce the first error message:
 At that time I was running 5 copies of reduce, each
 busy multiplying numbers for some hours. Right now
 I am only running the GHC by itself, so no other load.
 It is running for about 23 min now, the last 6 of which
 the memory usage stays at 15.9 Gb. My computer may
 have learned to increase the paging file on its own.
 The computer refuses to give up this time, but I can't
 wait, going to bed is long overdue, if it generates an
 error message I'll copy it in the morning. Well, memory
 usage decreased a bit to 15.1, this happened after it stayed
 at 15.9 for about 15 min. It I remember correctly what had
 happened the other day, it will increase again, and then give
 an error message. It's at 15.3 Gb for about 3 min now. On the
 other hand the processor is doing almost nothing, going between
 0% and 1%. Memory 15.4, 15.8, processor went up to 16% to 17%
 (which is = 1 out of 6 cores),
 memory 15.9, this is about 40 min after it started compiling
 and about 23 min after it went up to 15.9 Gb (15.8 presently).
 I should perhaps be patient and wait for the compiler to give up
 and give me an error message that I could copy, but I do not have
 the patience to do that: Again the original error message was
 something to the effect that the compiler refused to work
 because of insufficient amount of memory. I kept trying to
 compile, so apparently the compiler tried to be nice to me,
 and eventually compiled my program in 9 hours, after which
 when I tried to execute it, I was invited to write this report.
 It keeps compiling right now, at 16% (1 of 6 cores) processor,
 and 15.9 Gb memory usage, for about 47 min (30 min at 15.9).

 P.S. Thanks for Richard Eisenberg and Herbert Valerio Riedel
 for replying to my email. I initially had problems, trying
 to register and submit a new bug report, the BotScout
 stopped me, and it was too late at night to try to figure it,
 so I just emailed Richard in my frustration. Now, BotScout
 may not be the culprit: I did not read carefully what
 it asked me to do, and it was something like
 Evaluate `(!!6) ...'
 My knowledge of Haskell is rather superficial (I use it since it
 compiles and is relatively fast, and does not need an extra library
 to work with large numbers), so I have to often consult the
 tutorial for simple constructs. I have had some previous
 experience with Common Lisp (or Scheme, a course taken 12
 years ago) and in Lisp there is a command "evaluate", and
 also single quotation marks are used within the language
 occasionally. So, I just copied the entire line
 Evaluate `(!!6) (whatever it was)'
 and pasted it in the interpreter which gave me some
 nonsense which I sent back to BotScout. BotScout then
 gave me a second chance, and I do not quite remember
 if I tried to be more careful, perhaps not, so BotScout
 wouldn't even let me try anymore after that (even if
 I started from scratch ... perhaps it remembered me).
 After that happened I copied only the (!!6), etc, and
 the my GHC interpreter did evaluate something meaningful
 (a single number, no error message). I just didn't want
 to spend the time to think that late at night, and
 at first it seemed to me that the word Evaluate
 might have been part of a Haskell expression.
 I question if the above (perhaps too lengthy) report
 is of any use, but again, I follow the instruction,
 if in doubt, submit it. Thank you again!

--
Ticket URL: http://ghc.haskell.org/trac/ghc/ticket/9647
GHC http://www.haskell.org/ghc/
The Glasgow Haskell Compiler

GHC

GHC

GHC

GHC

GHC

GHC

GHC

GHC

tags

participants (1)