On Fri, 2009-06-26 at 13:00 +0100, Simon Marlow wrote:
Maybe bzlib allocates using malloc()? That would not be tracked by GHC's memory management, but could cause OOM.
Yes it does.
Another problem is that if you ask for a large amount of memory in one go, the request is usually honoured immediately, and then we GC shortly afterward. If this is the problem for you, please submit a ticket and I'll see whether it can be changed. You could work around it by calling System.Mem.performGC just before allocating the memory.
What I'd like is a way to inform the GC about how much memory there is attached to a ForeignPtr. The amount would contribute to the GC's decision on when to collect. This would solve the bzlib case and also the same problem in other bindings like cairo (big pixmaps) and gtk2hs (lots and lots of medium-sized foreign objects). Obviously it's not a total solution since not all bindings can easily discover the size of the foreign allocations, but in the case of zlib, bzlib, cairo and gtk2hs it would be possible. It also does not have to be 100% accurate. For implementation, perhaps keep in the rts an extra foreign memory count (which gets used by the GC in its accounting). Then on a foreign allocation we add to the count. An extra finaliser on the ForeignPtr could then decrement the foreign memory count. Another, more local, approach might be to keep the accounting info in a data structure directly, eg: data ForeignAccounting# which would contain a count. This type would be recognised by the GC. These could be embedded into data structures anywhere to account for the size of foreign allocations, in particular into the ForeignPtrContents. Duncan