On Mon, Jul 12, 2010 at 12:54 PM, Edward Z. Yang wrote:

Excerpts from Evan Laforge's message of Mon Jul 12 14:56:11 -0400 2010:

...

But I'm not convinced that's actually enough because the C code is still running outside of a withForeignPtr.  I would have to do something really hairy like call back to C from the haskell callback, wrapping it in withForeignPtr.  Is there a better way to convince the GC that the pointer should remain alive until I explicitly release it from C?  The ideal I think would be a ref count that would keep the pointer alive while >0, that could be decremented from C.

The function you are looking for is withForeignPtr.  touchForeignPtr also works.

Well, what I'm worried about is that withForeignPtr says you should only use the pointer from inside it. The situation here is that I've passed a pointer to C. C wants to share ownership of the pointer, so even if all haskell references are gone, it needs to stay alive until C says so. So there's nothing to pass to withForeignPtr. C uses a ptr, not a foreign ptr. My proposed solution is to stash a reference to the foreign pointer inside a callback in the hopes that will be treated as a GC root and keep it alive manually, until the callback is deleted manually with freeHaskellFunPtr. So the awkward version would be: - pass funptr haskell callback to C, that will return the desired ptr - C wants to use the ptr, so it calls back to haskell, with then calls back to C inside a withForeignPtr, passing the underlying ptr in - now the uses of the ptr are dynamically inside of a withForeignPtr - when C is done, it calls freeHaskellFunPtr on the funptr It makes the C awkward and you wind up with a stack that looks like haskell -> C -> haskell -> C. However, my suggested abbreviation is: - pass ptr to C directly, along with a funptr haskell callback to keep the ptr alive - C just uses the ptr directly, trusting that the funptr will keep it alive - when it's done, free the funptr as normal But where I'm unclear is what the funptr has to do to keep the ptr alive. Return it? Call touchForeignPtr on it? It seems weird to me because this funptr is never called, it's just there for the GC. Another hack I thought of: - store ptr in a global MVar in haskell, pass it to C - C does its thing, ptr stays alive because haskell still has a reference - now when C calls the finalize funptr, it deletes the ptr from the haskell MVar, which releases haskell's hold on it Effectively, this is using the global MVar as a GC root instead of the funptr itself. I'm much more confident that the GC is going to follow module level CAFs than some random funptrs allocated with a foreign "wrapper" call. However, the "wrapper" calls must serve as GC roots too, because otherwise what guarantees that variables captured in its closure stay alive? Right?

The easiest thing to do is copy the contents to a regular area of memory not managed by a Storable Vector.  This'll be much less painful because it's just a normal free (not a recursive one, which can get hairy).

Yeah, that's definitely the safest and simplest. But the copying defeats the purpose of passing a pointer in the first place, which was to not have to copy the giant array just to pass it.

On Mon, Jul 12, 2010 at 1:53 PM, Edward Z. Yang wrote:

Excerpts from Evan Laforge's message of Mon Jul 12 16:43:45 -0400 2010:

...

Yeah, that's definitely the safest and simplest.  But the copying defeats the purpose of passing a pointer in the first place, which was to not have to copy the giant array just to pass it.

Well, if your C code wasn't squirreling away the pointer, you could have avoided the copy.  Memory copies in C are suprisingly cheap; I'm really surprised how often I see memcpy() in high performance code (though, it may be the case that the details of the algorithm are a much more important influence on performance.)

From some quick tests, copying 32 32mb chunks takes 0.5 seconds, which is too much lag for interactive use, but only one is only around 0.01 seconds, which is not a problem. At the moment, I expect 32mb will be an upper bound, so I think I can get away with copying for now.

However, I'm still curious about the funptr approach. Does a "wrapper" funptr callback act as a GC root? On Mon, Jul 12, 2010 at 4:26 PM, Felipe Lessa wrote:

... I would try to avoid having the MVar/IOVar as a CAF because of its unpredictability.  You can always encapsulate it as somewhere else (e.g. Reader monad).

Yes, I would prefer a non-horrible hack too, but it's tricky and invasive to try to get a callback from C to haskell into the main haskell state monad. I would have to send a "delete this key" msg which the main loop will later pick up, which is certainly doable, but... invasive.

Excerpts from Axel Simon's message of Tue Jul 13 16:03:01 -0400 2010:

If your C code has a way to properly unref a pointer then you could wrap your ForeignPtr in a StablePtr and pass that to C land. Once C has freed the StablePtr the ForeignPtr can become dead when Haskell has dropped all references and it will be garbage collected.

Does ForeignPtr have a guaranteed representation from C land? I feel like you want to create a stable pointer and then unsafely cast the foreign pointer into a real pointer, which would effectively fizzle the original foreign pointer's finalizer. Cheers, Edward

Excerpts from Axel Simon's message of Tue Jul 13 16:28:29 -0400 2010:

Well, if the C code hangs on to the StablePtr that wraps the ForeignPtr, its finalizer won't be run. But can run again once the StablePtr is freed. So you can take out the Ptr in the ForeignPtr and use it in C land as long as C holds on to the StablePtr.

That's what I thought, just making sure. It occurs to me that there might be a long term memory leak here. Cheers, Edward

On 13/07/2010 05:49, Evan Laforge wrote:

On Mon, Jul 12, 2010 at 6:54 PM, John Meacham wrote:

...

Hi, is a StablePtr what you are after?

Indeed, it looks like StablePtr will get me what I want. It's a little less convenient than FunPtr because I'm already doing some finalization of FunPtrs and I can reuse the same callback, but it looks like it's specifically documented to do what I want, which is reassuring.

In any case, memcpy still wins out on simplicity. If I ever need to pass a pointer larger than 32mb and have both haskell and C manage the memory, I'll look into StablePtr.

FYI, when you call a foreign import "wrapper" to make a FunPtr, a StablePtr gets created behind the scenes to point to the closure representing the function in Haskell, and freeHaskellFunctionPtr calls freeStablePtr on that StablePtr. Rougly speaking, ForeignPtrs are pointers from Haskell to C, and StablePtrs are pointers from C to Haskell. However, we realised that we could optimise the case of a ForeignPtr allocated in Haskell by using heap-allocated memory and replacing the finalizer with the GC, which is what mallocForeignPtr does. Cheers, Simon