marlowsd:
I'll work with Simon to investigate the runtime, but would welcome any ideas on further speeding up cafe4.
An update on this: with the help of Alex I tracked down the problem (an integer overflow bug in GHC's memory allocator), and his program now runs to completion.
This is the largest program (in terms of memory requirements) I've ever seen anyone run using GHC. In fact there was no machine in our building capable of running it, I had to fire up the largest Amazon EC2 instance available (68GB) to debug it - this bug cost me $26. Here are the stats from the working program:
392,908,177,040 bytes allocated in the heap 174,455,211,920 bytes copied during GC 24,151,940,568 bytes maximum residency (6 sample(s)) 36,857,590,520 bytes maximum slop 64029 MB total memory in use (1000 MB lost due to fragmentation)
Generation 0: 62 collections, 0 parallel, 352.35s, 357.13s elapsed Generation 1: 6 collections, 0 parallel, 180.63s, 209.19s elapsed
INIT time 0.00s ( 0.11s elapsed) MUT time 1201.47s (1294.29s elapsed) GC time 532.98s (566.33s elapsed) EXIT time 0.00s ( 5.34s elapsed) Total time 1734.46s (1860.74s elapsed)
%GC time 30.7% (30.4% elapsed)
Alloc rate 327,020,156 bytes per MUT second
Productivity 69.3% of total user, 64.6% of total elapsed
Well done!!