October 2002 - FFI - Haskell.org

RE: C Struct Field Access
by Simon Marlow 17 Oct '02

17 Oct '02

> What's the best strategy for accessing fields in someone > else's C struct? > Should I write my own glue file with accessor functions? Or > should I make > a Storable instance for the struct? I wrote some stuff on accessing C structs from Haskell recently which might be useful: http://www.haskell.org/pipermail/ffi/2002-September/000720.html Cheers, Simon

1 0

RE: Finali[zs]ers
by Simon Peyton-Jones 17 Oct '02

17 Oct '02

John I'm not quite sure what you mean. I think you are saying that this is a reason it's important to have Haskell finalizers, but I'm not sure. How would you like to write a paragraph or two to explain the issue, and we can put it in the document? That way it would say what you mean. Simon | -----Original Message----- | From: John Meacham [mailto:john@repetae.net] | Sent: 14 October 2002 22:22 | To: ffi(a)haskell.org | Subject: Re: Finali[zs]ers | | I dont have cvs write access but you should mention something about | expressing gc dependencies between ForeignPtr's. the only two ways to do | this in the current haskell systems are | * weak pointers | * touchForeignPtr from the finalizer of another ForeignPtr | | if the second one is not allowed by restricting us to C only finalizers, | then the first needs to be added into the FFI, (or some other mechanism | invented). I consider this functionality vital for any non-trivial use | of ForeignPtr, and it becomes more important as we move more stuff to C | land, as C structures become more common. | | for more info on the use of this, see the FFI touchForeignPtr | documentation and the paper describing Weak pointers in ghc. the fact | that multiple people saw the need for this and came up with solutions | probably says something about its usefulness. | John | | On Mon, Oct 14, 2002 at 06:16:05PM +0100, Simon Marlow wrote: | > > Thanks to everyone who has contributed to this discussion so | > > far. Since | > > there are a lot of subtle points to be considered at once, I | > > thought I'd | > > try to write a summary of the issues raised so far. | > > | > > I'll put something in the repository so we can all hack on it. | > > Hopefully I'll have something ready later today. | > | > oops, I should have mentioned that if you just want to read the | > document, it will be available here: | > | > | > http://cvs.haskell.org/cgi-bin/cvsweb.cgi/haskell-report/ffi/finalizers. | > txt | > | > just as soon as the CVS web server updates its repository. | > | > Cheers, | > Simon | > | > _______________________________________________ | > FFI mailing list | > FFI(a)haskell.org | > http://www.haskell.org/mailman/listinfo/ffi | | -- | ------------------------------------------------------------------------ --- | John Meacham - California Institute of Technology, Alum. - john(a)foo.net | ------------------------------------------------------------------------ --- | _______________________________________________ | FFI mailing list | FFI(a)haskell.org | http://www.haskell.org/mailman/listinfo/ffi

2 1

Why I want Haskell finalizers
by George Russell 17 Oct '02

17 Oct '02

The document which Simon and Alastair have kindly put together asks for more details about why I want foreign garbage collectors to be able to call back to Haskell. Well I can't supply many details, because this is not immediately required by the UniForM/MMiSS project we are working on at Bremen. However the problem is a fairly simple and general one. If you have Haskell talking to some other language with a garbage collector, be it Java or SML, then at a given point of the program you can in general expect to have Haskell holding stable pointers to objects referenced from their world by the other language, and vice-versa. The most obvious way of garbage-collecting these is for each language to reference the foreign objects using its own version of ForeignPtr's, which then instruct the other's RTS that the corresponding stable pointer is no longer required. This mechanism does not handle circular pointers (Haskell references an ML object, which in turn references the original Haskell object again) but it is at least a start. Furthermore as a programmer I think my response to circular dependencies would be to try to avoid them, for example by breaking cycles by making some of the links weak, since a general solution would be fairly complex. I think in any case a general solution would very probably use finalizers to handle finalizers, while doing some complex pointer-chasing to detect and break cycles.

2 3

Finalizers compromise
by George Russell 17 Oct '02

17 Oct '02

I like what Simon and Alastair have written in the finalizers document. However I don't think there is any proposal I can wholeheartedly support, so I have to produce my own, which is a mixture of almost all the proposals (1)-(5). I hope though that this proposal will at least provide a framework which everyone can accept. The basic idea is that we accept that we cannot agree upon a final version of the FFI standard at this time, so we (a) make an interim change to the standard; (b) experiment with Haskell finalizers; (c) agree a method for settling discussion in a few months time. If anyone else has any better way of putting this discussion out of its misery, please let's hear it. Here is the proposal in detail: (1) We change the names of "newForeignPtr" and "addForeignPtrFinalizer" to "newUnsafeForeignPtr" and "addUnsafeForeignPtrFinalizer". This can be considered frozen and a settled part of the standard; people can start coding with them right away. (2) We also include functions newForeignPtr :: Ptr a -> IO () -> IO (ForeignPtr a) and addForeignPtrFinalizer :: ForeignPtr a -> IO () -> IO () However we should make it clear in the standard that (a) implementations only *may* choose to implement these functions, or may only implement them with restrictions; (b) the specification of these functions, or whether they exist at all, should be regarded by users as something that is particularly likely to change before the final cut. We can put this section of the standard in italics to indicate its provisional nature. (3) In the meantime Hugs, GHC, NHC should attempt to implement these functions (using for example Simon's patch) and we see how we get on, for example if there are more insuperable difficulties. (4) Also Hugs, GHC and NHC should implement PVars, with the interface newPVar :: a -> IO (PVar a) updatePVar :: PVar a -> (a -> (a,b)) -> IO b and we can see if they adequately address Alastair's concerns about mutable state (unless he already considers them adequately answered). Of course for Hugs and GHC the implementation is completely trivial, and if you can't find a student with 10 minutes to spare to write them in terms of MVars, I will be happy to oblige . . . NB. The question is not here "Are PVars the best/only possible implementation of concurrency", just "Do they provide a useful and implementable way for Haskell finalizers to communicate". NB2. If Alastair insists on "blockFinalizers" instead, I don't think we need to argue the point, but I would object to having "blockFinalizers" in the FFI standard, since blockFinalizers seems to me to make too many assumptions about the nature of the implementation (that state changes by Haskell have some sort of universal ordering for example). I would also object to having PVars in the FFI standard, since there may be a better way of implementing thread-safe common state. (5) Since perhaps we will not have unanimity even if we do all this and come back in a year, perhaps we should all agree now some decision procedure. My suggestion is that we agree now that we will either (a) cast newForeignPtr and addForeignPtrFinalizer out of the standard altogether; or (b) make them compulsory or (c) settle some alternative specification/type for them when at least two of {the NHC team}, {the Hugs team}, {the GHC team} consent. Hopefully this will not take too long, perhaps about 6 months.

1 0

Finalizers Ride Again
by George Russell 16 Oct '02

16 Oct '02

Malcolm Wallace wrote (about PVar's, my proposed interface for useful mutable state in such a way that they can be implemented in NHC, GHC and Hugs): [snip] > I'll need to think some more about the details of PVars, but my first > reaction is that they would very easy to implement. However, I fear it > may be no good persuading me of their value if you cannot also persuade > the other implementations to adopt them, so I'd like to hear what Hugs > and ghc people have to say, especially whether they would also solve > the problems with mutable shared state raised by Alastair. [snip] I don't see that it's necessary for us to come to a decision right now about PVars unless we want to put them in the FFI standard. Well we could do that, in the same way as we have unsafePerformIO, if it's what it takes us to see the back of this finalizers discussion, but I don't think it's particularly necessary; although I think it might be a good idea to have a standard definition of something like PVars so that people can do useful manipulations of mutable state in finalizers, such things don't really seem to me to belong in an FFI standard. What's important now is that I think I've proved the point that there is an interface to mutable state which can be manipulated usefully and safely from Haskell finalizers in all of Hugs, GHC and NHC. Also, in view of Simon Marlow's patch and what Malcolm Wallace has said, it looks as if Haskell finalizers are at least implementable by all three. Therefore I see very little left to stop them going back into the FFI standard. I'm about to go on holiday for several days, so I look forward to coming back to find the whole issue safely buried in the FFI archives. 8-)

4 8

RE: The Revenge of Finalizers
by Simon Marlow 16 Oct '02

16 Oct '02

> > So, are we now claiming that my patch *is* safe? (Never mind about > > IORefs, I'm talking about the implementation itself). > > No. Then I'm confused. Allow me to quote your previous message: > Most calls to IO actions from primitives are safe > and I believe these ones are too. So is it just that there are a small number of places where it is unsafe to invoke IO from a primitive, or is there something else going on? Can we easily identify which are the unsafe places and fix them? Cheers, Simon

2 1

RE: The Revenge of Finalizers
by Simon Marlow 15 Oct '02

15 Oct '02

> > Also, this is a nested call to eval(), in a primitive, which can > > invoke an IO action and therefore re-enter Haskell without going > > through unsafePerformIO. Is that safe? > > Yes, I think so. Most calls to IO actions from primitives are safe > and I believe these ones are too. (In some ways, its the rarity of > race conditions, etc. biting you that disturbs me most about Haskell > finalizers: it's hard to remember to do the right thing when the wrong > thing happens so rarely.) So, are we now claiming that my patch *is* safe? (Never mind about IORefs, I'm talking about the implementation itself). Cheers, Simon

2 1

RE: The Revenge of Finalizers
by Simon Marlow 15 Oct '02

15 Oct '02

> > Indeed, I very nearly implemented such a thing as part of the patch > > I sent out. However, it didn't look trivial enough to implement so > > I backed off. The "blocked" state needs to be saved & restored at > > various points: when starting a finalizer, when invoking a > > foreign-exported function, and when invoking an exception handler. > > I think it's fairly easy. A sketch is something like: > > // primitive called from Haskell > void blockFinalizer(Cell m, State world) { > blocked += 1; > eval(ap(m,world)); > blocked -= 1; > runFinalizers(); > } Does this run into problems with exceptions? Also, this is a nested call to eval(), in a primitive, which can invoke an IO action and therefore re-enter Haskell without going through unsafePerformIO. Is that safe? Cheers, Simon

2 1

RE: Finali[zs]ers
by Simon Marlow 15 Oct '02

15 Oct '02

> I dont have cvs write access but you should mention something about > expressing gc dependencies between ForeignPtr's. the only two > ways to do > this in the current haskell systems are > * weak pointers > * touchForeignPtr from the finalizer of another ForeignPtr Actually I have a feeling that in GHC it doesn't work to use touchForeignPtr from the finalizer of one ForeignPtr in order to keep another alive. According to our semantics for weak pointers, I think the two finalizers can still be run together, because pointers from a finalizer are not followed when calculating reachability. Hmm. I'm not sure what to do about this. > if the second one is not allowed by restricting us to C only > finalizers, > then the first needs to be added into the FFI, (or some other > mechanism > invented). I consider this functionality vital for any non-trivial use > of ForeignPtr, and it becomes more important as we move more > stuff to C > land, as C structures become more common. Yes, this is a good point. I'll add it to the document. > for more info on the use of this, see the FFI touchForeignPtr > documentation and the paper describing Weak pointers in ghc. the fact > that multiple people saw the need for this and came up with solutions > probably says something about its usefulness. Cheers, Simon

1 0

RE: [Alastair Reid <alastair@reid-consulting-uk.ltd.uk>] Re: cvs commit: haskell-report/ffi finalizers.txt
by Simon Marlow 15 Oct '02

15 Oct '02

> Things you mights add are: > > eg(1) in composite finalizers: > > This would be handled just fine by C finalizers if > addForeignPtrFinalizer guaranteed to execute finalizers in reverse > order of their addition. This seems like a reasonable thing to > specify since addForeignPtrFinalizer is much less useful if this > guarantee doesn't hold and because it is easy to implement. Ok, I've added that. > eg(2): > > isn't too convincing since you could easily write the same thing in > C. I expect you could find something more convincing with a little > effort. Please feel free to suggest something - I can't off the top of my head think of anything that you couldn't also do in C, but then that's not the point of the exercise. The point is that it's rather more convenient in Haskell. > Communicating with a foreign runtime/garbage collector: > > I remain unconvinced that this example has been worked out in detail > and believe that if it were worked out in detail we'd either find > that all George needs is to be able to call into the _runtime > system_ during a Java GC (doesn't require Haskell finalizers) or > that he requires a much more intimate relationship with Haskell's GC > (requires a lot more than the FFI provides). I'll let others comment on this. > Other cleanups: > > 1) > Unsavouriness of exporting &free seems like a judgement an impartial > witness would avoid. Well, at some point we have to make judgements about what is a good design. I assumed this was an uncontested point - is it? > Manipulating mutable state and communicating with other threads: > > I don't see how adding concurrency strengthens the argument for > Haskell finalizers. Well, when you add concurrency Haskell finalizers become much more useful because you can manipulate Haskell-side mutable state using them. Without concurrency (or perhaps a blockFinalizers primitive), you can't. We're taking slightly different viewpoints here: you're saying that we must have mutable state and Haskell finalizers are useless without it, whereas I'm saying that Haskall finalizers are still useful even if you can't manipulate Haskell state inside one. Essentially this comes down to a matter of opinion, and we won't reach a conclusion without more evidence. > Adding mutable Haskell state doesn't seem to strengthen it either. > Indeed, I assume the existence of mutable Haskell state in arguing > against Haskell finalizers. And that's exactly the standpoint I disagree with. The document goes to some lengths to consider the utility of Haskell finalizers in the absence of mutable state. > Arguments against: > > Adding Haskell finalizers drags in many concurrency issues for > programmers: > > - potential to introduce race conditions > - potential for deadlock These points are made later in the document, and the "arguments against" section forward-references them. > As with most concurrency issues, the resulting bugs are easy to > introduce, hard to reproduce and hard to track down. If you use IORefs, then you have the same concurrency bugs in GHC as you would in Hugs, so I don't believe that argument. Most programmers will understand the concept of a race condition. So then it remains to explain to them that if they want to use IORefs from finalizers, they must use GHC and MVars instead. > Adding Haskell finalizers has a high implementation burden for those > who have not already implemented preemptive concurrency - greatly > increasing the complexity of implementing Haskell + FFI (+ mutable > state, of course - but that is a trivial thing to implement). I think we've already demonstrated that the implementation burden isn't anything like as bad as we thought it was. > The complexities introduced seem out of line with the problem being > solved. > > > Implementations/Hugs: > > You say: "There is some doubt over whether the patch is > safe as it stands:" > I say: "There is no doubt that the patch is unsafe as it stands:" Ok, I'll change the wording. How about checking the safety conditions? > Implications of allowing Haskell finalizers in non-concurrent > systems/Mutable state: > > You omit my argument that, since the primary goal of finalizers is > to modify mutable state, Haskell finalizers which do not modify > mutable Haskell state will be of little use. Ok, so your opinion is that Haskell finalizers are useless unless you can modify mutable Haskell state. The document gives some rationale as to why Haskell finalizers might still be useful, but again this comes down to a matter of opinion. I've added the following point to the "arguments against" section: 3. If finalizers cannot modify mutable Haskell state (see later), then they are significantly less uesful in a system that supports mutable Haskell state, and we might prefer to avoid the complexity on the grounds that the payoff isn't worth it. > All they will be able > to do is lookup objects in immutable Haskell state and invoke C > finalizers to do their work - a significant limitation. (In > fairness, I will say that useful information might be stored in the > finalizer closure so actions like looking up an object in a binary > tree of all live objects (say) might be avoidable with a little > effort.) These points are already made in the document. > Interaction with co-operative concurrency: > > You say (though your final paragraph suggest you don't > insist on it): > > One might argue that since you have to do this anyway in Hugs - > another thread can only get control at an explicit yield point - > doing it for finalizers too isn't so bad. > > I say: > > There is a world of difference between being delayed for an > unknown, finite amount of time and being delayed for an unbounded, > potentially infinite amount of time. (This is why vague concepts > like 'fairness' are useful in reasoning about scheduling - you > don't know when you will run but you know you will get a shot > sometime.) I think these are completely unrelated points. In fact, I agree with your statement, but it doesn't describe the behaviour of finalizers: currently there's no guarantee that finalizers are run in a finite amount of time. There is a qualitative difference here, but we have to be very careful about how it is described. One thing you can say is that a program written in a certain way in a co-operatively scheduled system can force finalizers to be delayed until the end of the program, whereas the same should not be true of a pre-emptively scheduled system. > Proposals: > > Add: > > 4) No change to the spec for the time being, while we evaluate the > need and cost of implementing Haskell finalizers in a range of > implementations. Note that upgrading from C finalizers to > Haskell finalizers would only require that we say that finalizers > are invoked with the equivalent of a 'safe' ffi call instead of > an 'unsafe' ffi call and, no doubt, the addition of some > convenience functions. > > I don't think this is the same as (3) Ok. Cheers, Simon

1 0