Hi Alastair, Thanks for the concise summary of the problem raised by shared thunks, and for the "keepAlive" proposal.

[btw will you be in New Haven around 16-19 Nov? I'm going to swing through there on my next trip over and it'd be good to see you and maybe ever humiliate myself again in the Gunks with you.]

[yes, I'll be around then, and look forward to much mutual humiliation during your visit! :)]

I think it was agreed that we need to replace touchForeignPtr with something else to let us express liveness dependencies. I hope you'll take part in that discussion since you and John Meacham are the only ones who seem to have used it so far.

Gladly. You indicated that you were somewhat unclear why we need liveness dependencies. I'll attempt to clarify by sketching some of the details of the particular C library for which I am writing FFI wrappers. I have a C library for 2D vector graphics. Two of the abstract types provided by this C library are: Pixmap -- A handle to an actual buffer of raster data RenderContext -- A handle that encapsulates all state associated with rendering, such as the current color, current font, target pixmap, etc. Note that it is possible to create many RenderingContext's that all render on to the same underlying Pixmap. To see why we need liveness dependencies, consider the following typical usage scenario in Haskell: do pm <- createPixmap -- 1 rc <- createRenderContext pm -- 2 drawBox rc -- 3 ... Note that, in the above, it's possible that the call to createRenderContext in line 2 could be the last Haskell reference to pm, making it a candidate for collection. But we don't actually want the Pixmap to be collected (and its finalizer invoked) until both the Pixmap *and* all associated rendering contexts which refer to the Pixmap become unreachable. The reason we need liveness dependencies is because, internally, the RenderContext maintains a pointer to the target Pixmap. But because this pointer exists only in the C heap, we need some way to inform Haskell's garbage collector that whenever a particular RenderContext is reachable, then its target pixmap is also reachable.

As a strawman to get discussion rolling, would something like the following do the job?

Yes, almost. See below for details.

-- | -- keepAlive x y ensures that the finalizer for y is not run -- until after the finalizer for x has run to completion. -- Of course, it might not even run then if y is still live -- at that point. keepAlive :: ForeignPtr a -> ForeignPtr b -> IO ()

Off the top of my head, I'd say the name sucks, the argument order is open to change and the semantics will cause headaches for GHC. Any other objections? :-)

Henrik Nilsson and I had a good discussion about this over lunch, particularly the subsequent issue about whether or not: keepAlive p1 p2 >> keepAlive p2 p1 should create un-collectible garbage. Our conclusion was that "finalization order" is a red herring, and that this primitive should not try to make any kind of guarantees about finalization order. All we need is some way to convey to Haskell's collector that liveness of one particular ForeignPtr implies liveness of another. I like your suggested "keepAlive" name, so I'd propose the following: -- | -- keepAlive x y informs Haskell's garbage collector that if x -- is live, then y is also live. keepAlive :: ForeignPtr a -> ForeignPtr b -> IO () Note that I have deliberately omitted any mention of the order in which the finalizers are invoked. Note, too, that in this version of keepAlive, it is perfectly reasonable to do something like: keepAlive p1 p2 >> keepAlive p2 p1 This will result in a cyclic structure that will be collected just like any other cyclic structure, and p1 and p2's finalizers will both be invoked, in some indeterminate order. I can't speak for other users who might need some guarantees about finalization order; I simply claim that the above definition is: (a) adequate for capturing foreign liveness dependencies, and (b) allows for ForeignPtr's to objects that may have cyclic references in their foreign representation. I think that (b) is more important than guaranteeing any particular finalization order, and that committing to a particular finalization order is both difficult to implement and not very robust.

...

I would really like to have just a little bit more in-depth understanding of why Haskell finalizers are an impossibility. In particular, the current finalizers.txt document on cvs.haskell.org states:

[...]

Thank you very much for the clear, articulate summary. I think I understand the issue(s) a bit better now. -antony -- Antony Courtney Grad. Student, Dept. of Computer Science, Yale University antony@apocalypse.org http://www.apocalypse.org/pub/u/antony

Hi Manuel, Sorry for the delay in replying. I'll quote a little more of the previous messages than usual to help refresh context. Manuel M T Chakravarty wrote:

Antony Courtney <antony@apocalypse.org> wrote,

...

You indicated that you were somewhat unclear why we need liveness dependencies. I'll attempt to clarify by sketching some of the details of the particular C library for which I am writing FFI wrappers.

I have a C library for 2D vector graphics. Two of the abstract types provided by this C library are: Pixmap -- A handle to an actual buffer of raster data RenderContext -- A handle that encapsulates all state associated with rendering, such as the current color, current font, target pixmap, etc.

Note that it is possible to create many RenderingContext's that all render on to the same underlying Pixmap.

To see why we need liveness dependencies, consider the following typical usage scenario in Haskell: do pm <- createPixmap -- 1 rc <- createRenderContext pm -- 2 drawBox rc -- 3 ...

Note that, in the above, it's possible that the call to createRenderContext in line 2 could be the last Haskell reference to pm, making it a candidate for collection. But we don't actually want the Pixmap to be collected (and its finalizer invoked) until both the Pixmap *and* all associated rendering contexts which refer to the Pixmap become unreachable.

The reason we need liveness dependencies is because, internally, the RenderContext maintains a pointer to the target Pixmap. But because this pointer exists only in the C heap, we need some way to inform Haskell's garbage collector that whenever a particular RenderContext is reachable, then its target pixmap is also reachable.

IMHO you are trying to compensate for a flaw in the whole setup:

* Line 1: You get a pointer to a C object assuming it is the last reference to that C object.

* Line 2: You pass this pointer back to C without copying it; ie, the only reference to the C object is in C land.

At this moment, the pointer obtained on Line 1 is no longer the business of the Haskell system. It is a pointer in C land to a C object; so, memory management of that structure should be let to the C library.

I'm sorry, but I simply don't agree with your rationale here (nor do I see a "flaw in the whole setup"). Yes, your observations about when references are live in C and when references are live in Haskell in the above code fragment is correct. However, in my opinion, this is an implementation detail. The user of my Haskell library should not know or care whether the library is implemented in Haskell, in C, or in some combination of the two. In this case, Pixmap and RenderContext could very easily be implemented entirely in Haskell (i.e. just make Pixmap a byte array, and RenderContext a record type that maintains a Pixmap in one of its fields). If it were implemented this way, then of course any live reference to a RenderContext will ensure that the Pixmap it refers to will not be GC'ed, since the field of the RenderContext record would contain a reference to the Pixmap. I see liveness dependencies as a way for me (as a library implementor) to use an external (C language) representation for a Haskell data structure, whilst retaining one of the most important benefits of programming in Haskell (garbage collection).

Assume the following C function

RenderContext *createPixmapWithContext () {

Pixmap *pm = createPixmap (); return createRenderContext (pm); }

in conjunction with

do rc <- createPixmapWithContext drawBox rc

How is this different from your Haskell code in a way that requires a foreign pointer dependency in one case, but not in the other?

For starters, I would never, ever write the createPixmapWithContext() function in C because it is an obvious memory leak. It allocates two objects (via createPixmap and createRenderContext), but returns a pointer to only one of them. You could potentially get away with this if you happen to use some reference counting scheme in C, but I never suggested I was doing any such thing (more on this below). To be honest, I don't really see your point here. I am implementing a Haskell library that happens to use some external (C language) representations for some data structures, and I would like to use Haskell's garbage collector to ensure that this Haskell library works as a Haskell programmer would expect. What you have presented above is an arbitrary C function that does some heap allocation that is never visible to the Haskell runtime. I would never expect liveness dependencies (or anything else) to enable the Haskell runtime to track heap allocation in arbitrary C code.

[...] As `createPixmapWithContext()' demonstrates, C land must free `pm' when the last render context referring to `pm' dies.

Not necessarily true! What you are suggesting (reference counting) is one possible memory management strategy in C, but is by no means the only option. Another possibility (the one I actually use) is simply for RenderContexts to do absolutely no memory management of the underlying Pixmaps whatsoever. Then it is up to whoever created the Pixmap to free the Pixmap, and up to whoever created a RenderContext to ensure that the RenderContext will not be used after its underlying Pixmap has been freed. This is relatively easy to document in prose in a library manual page. I see liveness dependencies as a way of exporting exactly such informal requirements to a high-level language's garbage collector. For C libraries I prefer this kind of explicit memory management scheme over reference counting because: (a) it is simpler to implement, (b) it provides a foundation for implementing higher level memory management schemes in C (it easy to wrap these primitive objects in higher level constructs that provide reference counting or arena-based allocation if you want that) and (c) the library can be exported to a garbage collected language, without any "impedance mismatch" between reference counting on the C side (with its known flaws collecting cyclic structures) and the calling language's GC scheme.

IMO the only clean way to approach this problem is to add a reference counting scheme to `pm' in C land.

Obviously I disagree. I hope I've clearly articulated why, and that there is a reasonable, simple alternative.

BTW, this is exactly how this problem is solved in the GTK+ GUI toolkit.

Reference counting is a decent crutch (with some known flaws) for those who are stuck programming in C. But for those of us working with a true high level language, I think using foreign pointers, finalizers and liveness dependencies to enable use of the high level language's collector when programming in the high level language is a far better alternative. -antony -- Antony Courtney Grad. Student, Dept. of Computer Science, Yale University antony@apocalypse.org http://www.apocalypse.org/pub/u/antony

John Meacham <john@repetae.net> writes:

here are my canidate suggestions:

* add a subset of Weak pointers

Can you spell this out in detail. What are the functions? What are their semantics?

- or -

* add addForeignDependency :: ForeignPtr a -> ForeignPtr b -> IO ()

Again, could you spell out the semantics.

note that breaking these dependencies might be tricky/impossible

This is something I don't understand. Why do you have to break dependencies? Is it to break dependency cycles? Also, when do you break dependencies? e.g., Do C finalizers need a way to tell Haskell to break a dependency? Do you manually detect that some objects have not been collected and speculatively start breaking dependencies? If the problem is cycles, would it be helpful if addForeignDependency detected when it created a cycle and raise an exception/ set a flag/ etc. -- Alastair