September 2002 - FFI - Haskell.org

RE: addForeignPtrFinalizer
by Simon Marlow 26 Nov '02

26 Nov '02

> > Actually in the current implementation the finalizers are not > > attached to the ForeignPtr at all: it's the other way around. > > Ah, I see. > > Fortunately, I don't think my sketched implementation depended on such > an attachment since it added an attachment of its own. The design is > intended to sit on top of the current GHC implementation (i.e., one > with Haskell finalizers and no control over the order they are run). Yes, you're right. We could do it that way. I'm also concerned about adding the extra overhead though, especially if this is something that will rarely be needed. Are we sure this is what we want? At least I can optimise mallocForeignPtr again and avoid registering the finalizer until the first call to addForeignPtrFinalizer. Cheers, Simon

3 4

RE: addForeignPtrFinalizer
by Simon Marlow 03 Oct '02

03 Oct '02

> > Then I'll reformulate my question as a patch. [...] > > Is there anything fundamentally wrong with this approach? > > I still maintain that getting this to work, testing it and, > especially, maintaining it is a lot of work. Isn't it worth a little effort? Requiring finalizers to be foreign functions is a serious wart in the FFI, especially when we're strongly advocating that as much marshalling should be done in Haskell as possible. As for the testing and maintenance, you can see from the patch that the changes are quite small and mostly obviously correct. The only difficult correctness property is whether it's ok to have eval() invoke runIO() and thereby eval() recursively, but the reasoning is simple: from eval() you can already call a foreign function, and foreign functions can invoke foreign-exported Haskell functions, which in turn re-enter eval() through runIO(). So the patch doesn't add any new invariants. You do have to worry about the impact of running Haskell finalizers at any point during execution of Haskell code. Since you can't do any concurrency synchronisation between the finalizer and the main Haskell thread, shared access to mutable data is impossible. This is just something you'd have to document carefully (note, it's not something you can do with the current finalization mechanism either, but there it's obvious). > [For example, you > mention that finalizers that point to the object they finalize don't > work. I guess we could fix that by adding GHC-style WeakPtrs.] This isn't as big a problem as it sounds. The finalizer can refer to the Ptr, just not the ForeignPtr. If it is essential to refer to the ForeignPtr (I can't think of a reason why it should be), then we could change the spec so that it was passed as an argument to the finalizer. Cheers, Simon PS. I'm sorry to keep banging on about this. Ultimately it doesn't really matter to me that much, since I only really use mallocForeignPtr. I guess I was just intrigued to see if the problem was really as difficult as we'd thought.

3 9

Re: FFI digest, Vol 1 #218 - 3 msgs
by George Russell 02 Oct '02

02 Oct '02

Simon Marlow wrote [snip] > Isn't it worth a little effort? Requiring finalizers to be foreign > functions is a serious wart in the FFI I agree, as I've stated before. It's not just that it's an irritating restriction, it's also because it looks just the sort of thing that will bite, and bite badly, when people start implementing non-trivial systems where multiple GCs are operating. I've already had far too much of this sort of thing. Tracking down the causes of unreproducible coredumps caused by mysterious interactions between the scheduler, the garbage-collector, and external programs, is not exactly the sort of thing I enjoy doing. [snip] > PS. I'm sorry to keep banging on about this. Ultimately it doesn't > really matter to me that much, since I only really use mallocForeignPtr. > I guess I was just intrigued to see if the problem was really as > difficult as we'd thought. [snip] I'm glad you are banging on about it. But since it looks unlikely that there is going to be agreement on the matter, I suggest the following compromise. We remove the restriction that finalizers may not call Haskell functions in the main body of the standard, but add a note that says that implementations may opt to impose such a restriction, provided they document it in some standard way.

3 4

RE: addForeignPtrFinalizer
by Simon Marlow 30 Sep '02

30 Sep '02

> > Still hoping ;-) The discussion seemed to stop without reaching a > > conclusion last time. > > I thought it was concluded and the report changed such that all three > compilers which implement the ffi spec can implement it without > receiving a heart, lung and liver transplant. Then I'll reformulate my question as a patch. Enclosed is a proof-of-concept patch that implements Haskell finalizers in Hugs, which works on some simple tests that I've run. There's nothing particularly clever going on - I'm just suspending execution of the finalizers until the next call to eval(). Is there anything fundamentally wrong with this approach? Possible problems I can think of: - It doesn't work too well with Hugs's cooperative multitasking: don't try to synchronise with another thread from a finalizer. The same problem occurs with foreign calls which re-enter haskell. - It adds a small amount of overhead to eval(); I haven't measured it, but I'm guessing it's not enough to worry about. - If you refer to the ForeignPtr from the finalizer, then the garbage collector will never discover the ForeignPtr is dead. There are GC techniques to avoid this problem, though. I've enclosed a simple test which runs finalizers within finalizers down to an arbitrary depth. Cheers, Simon

3 2

RE: addForeignPtrFinalizer
by Simon Marlow 27 Sep '02

27 Sep '02

> On Fri, Sep 27, 2002 at 12:25:31PM +0100, Simon Marlow wrote: > > Then I'll reformulate my question as a patch. Enclosed is a > > proof-of-concept patch that implements Haskell finalizers > in Hugs, which > > works on some simple tests that I've run. There's nothing > particularly > > clever going on - I'm just suspending execution of the > finalizers until > > the next call to eval(). > > It seems to have problems if more than one finalizer is > pending, probably > because runIO() calls eval(): > > t = do > sequence_ (map f [1..30]) > performGC > where f n = do > p <- mallocBytes 64 > newForeignPtr p $ do > free p > putStrLn ("finalize " ++ show n) Looks like I missed out a pop() - the enclosed patch works with your example. (I'm not very familiar with the internals of Hugs). Cheers, Simon

1 0

RE: addForeignPtrFinalizer
by Simon Marlow 27 Sep '02

27 Sep '02

> > I tell you what, I'll implement finalizer ordering if you > guys implement > > full Haskell finalizers :-) > > How about providing newForeignPtr as specified? (Or are you > still hoping?) Still hoping ;-) The discussion seemed to stop without reaching a conclusion last time. At least, I still don't think I fully understand why the current newForeignPtr is not implementable without concurrency, given that a form of context switching must already exist in order to implement foreign import that re-enters Haskell. Anyway, it's not that big a deal. I'll hopefully get round to making the change soon. Cheers, Simon

3 3

Re: ANN: H98 FFI Addendum 1.0, Release Candidate 7
by Ross Paterson 26 Sep '02

26 Sep '02

Alastair Reid <alastair(a)reid-consulting-uk.ltd.uk> wrote: > I guess the issue is that if someone wanted to use MarshalAlloc.free > as a finalizer they would not be able to do so. Since we don't > guarantee that MarshalAlloc.malloc is "stdio.h malloc", they couldn't > portably cons up a compatible &free. Yes, you're trying to recover something that was lost with the change to newForeignPtr. Formerly one could allocate something with malloc or mallocArray and add a finalizer that called free. Now you can't, except in the canned special case of mallocForeignPtr. But there's nothing similar for arrays, as required e.g. in Data.Array.Storable.

3 4

RE: addForeignPtrFinalizer
by Simon Marlow 25 Sep '02

25 Sep '02

> > However, to provide a general ordering property on finalizers would > > be quite awkward in GHC - I just can't think of an easy way to do > > it. [Awkward ways deleted] > > If I understand correctly, the problem is that each FP has a cloud of > finalizer objects attached to it with no particular structure between > finalizer objects and the reason for this unstructured cloud > is because you're piggy-backing on some other mechanism. Actually in the current implementation the finalizers are not attached to the ForeignPtr at all: it's the other way around. This paper explains the details: http://www.haskell.org/~simonmar/papers/weak.ps.gz There's currently no way to get from a ForeignPtr to its list of finalizers (well, you could do a full GC and figure it out). Cheers, Simon

2 1

RE: addForeignPtrFinalizer
by Simon Marlow 25 Sep '02

25 Sep '02

> On Wed, Sep 25, 2002 at 09:33:04AM +0100, Ross Paterson wrote: > > If the struct were allocated with mallocForeignPtr, you need another > > C call to use as the finalizer for that, say hs_f_free(). > > Sorry, that's wrong: If the struct were allocated with > mallocForeignPtr, > you're stuck. You've no choice but to use addForeignPtrFinalizer, > except that you can't because the order of finalizers is unspecified. I can guarantee in GHC that if you add a finalizer to a ForeignPtr made with mallocForeignPtr, then it will run before the memory is released. However, to provide a general ordering property on finalizers would be quite awkward in GHC - I just can't think of an easy way to do it. One way is to add a unique tag to each weak pointer object and sort them before running the finalizers, but also the finalizers have to run sequentially somehow (currently there's a hack to run some finalizers in the same thread, but it's nothing more than a performance hack; I don't think I could use it to provide reliable ordering). I tell you what, I'll implement finalizer ordering if you guys implement full Haskell finalizers :-) Cheers, Simon

3 2

addForeignPtrFinalizer
by Ross Paterson 25 Sep '02

25 Sep '02

The spec says there are no guarantees on the order in which the finalizers are run. Doesn't this make this function almost impossible to use? Suppose one finalizer frees the storage and the other cleans up something it refers to. I'd suggest running the finalizers in the reverse order to the order they were added, i.e. the new finalizer will be run before any existing ones. Now that finalizers are FunPtrs, there should be fewer semantics problems.

2 6