The "hook" idea works with static linking: the RTS provides a default version of the hook, that can be overriden by a user-supplied function of the same name. This is what GHC does. However, our dynamic linker doesn't support this kind of overriding. The system's dynamic linker does, though: that's why you can still provide your own malloc() and functions in libc.so will use yours rather than the default one.
Note that the Darwin and (AFAIK) Windows dynamic linker do not support this behaviour. They explicitly avoid that kind of behaviour to prevent accidental overriding. (What happens on Linux if you link a GHC-compiled program to a shared library that internally uses a function named "allocate"? What if the next revision of that library introduces such a function?) What are the alternatives to using these hook functions? Explicitly looking for a OutOfHeapHook symbol in the executable using dlsym and friends? Exporting a "RegisterOutOfHeapCallback" function from the rts? Both seem a bit inconvenient to me, but some change might be necessary when we use dylibs/dlls. Cheers, Wolfgang