There's something strange going on in this example. For instance, setting (-M) heap limits as low as 40K seems to have no effect, even though the program easily uses more than 8G. Except, interrupting the program in such a case does seem to give a message about heap limits being exceeded (it won't stop on its own, though). Also, the program compiled without optimizations uses very little memory (though it's slow), which is odd, because the optimized version works exclusively on Int#, which shouldn't cause heap allocation.

I filed a ticket earlier: http://hackage.haskell.org/trac/ghc/ticket/7850