RE: addForeignPtrFinalizer
Then I'll reformulate my question as a patch. [...] Is there anything fundamentally wrong with this approach?
I still maintain that getting this to work, testing it and, especially, maintaining it is a lot of work.
Isn't it worth a little effort? Requiring finalizers to be foreign functions is a serious wart in the FFI, especially when we're strongly advocating that as much marshalling should be done in Haskell as possible. As for the testing and maintenance, you can see from the patch that the changes are quite small and mostly obviously correct. The only difficult correctness property is whether it's ok to have eval() invoke runIO() and thereby eval() recursively, but the reasoning is simple: from eval() you can already call a foreign function, and foreign functions can invoke foreign-exported Haskell functions, which in turn re-enter eval() through runIO(). So the patch doesn't add any new invariants. You do have to worry about the impact of running Haskell finalizers at any point during execution of Haskell code. Since you can't do any concurrency synchronisation between the finalizer and the main Haskell thread, shared access to mutable data is impossible. This is just something you'd have to document carefully (note, it's not something you can do with the current finalization mechanism either, but there it's obvious).
[For example, you mention that finalizers that point to the object they finalize don't work. I guess we could fix that by adding GHC-style WeakPtrs.]
This isn't as big a problem as it sounds. The finalizer can refer to the Ptr, just not the ForeignPtr. If it is essential to refer to the ForeignPtr (I can't think of a reason why it should be), then we could change the spec so that it was passed as an argument to the finalizer. Cheers, Simon PS. I'm sorry to keep banging on about this. Ultimately it doesn't really matter to me that much, since I only really use mallocForeignPtr. I guess I was just intrigued to see if the problem was really as difficult as we'd thought.
"Simon Marlow" <simonmar@microsoft.com> wrote,
Then I'll reformulate my question as a patch. [...] Is there anything fundamentally wrong with this approach?
I still maintain that getting this to work, testing it and, especially, maintaining it is a lot of work.
Isn't it worth a little effort? Requiring finalizers to be foreign functions is a serious wart in the FFI, especially when we're strongly advocating that as much marshalling should be done in Haskell as possible.
As for the testing and maintenance, you can see from the patch that the changes are quite small and mostly obviously correct. The only difficult correctness property is whether it's ok to have eval() invoke runIO() and thereby eval() recursively, but the reasoning is simple: from eval() you can already call a foreign function, and foreign functions can invoke foreign-exported Haskell functions, which in turn re-enter eval() through runIO(). So the patch doesn't add any new invariants.
You do have to worry about the impact of running Haskell finalizers at any point during execution of Haskell code. Since you can't do any concurrency synchronisation between the finalizer and the main Haskell thread, shared access to mutable data is impossible. This is just something you'd have to document carefully (note, it's not something you can do with the current finalization mechanism either, but there it's obvious).
I have to say that, given Simon's patch, I am inclined to revert back to the old API for foreign pointers. The restriction on pure C land finalizers *is* awkward, and as we have already seen implies further changes (ie, adding something like `finalizerFree'). It surely would have been too much to require a Haskell system to implement pre-emptive concurrency just to support finalizers, but a change along the lines of Simon's proposal doesn't seem to be too much trouble for a much nicer design (which I regard Haskell land finalisers to be).
[For example, you mention that finalizers that point to the object they finalize don't work. I guess we could fix that by adding GHC-style WeakPtrs.]
This isn't as big a problem as it sounds. The finalizer can refer to the Ptr, just not the ForeignPtr. If it is essential to refer to the ForeignPtr (I can't think of a reason why it should be), then we could change the spec so that it was passed as an argument to the finalizer.
I'd be perfectly happy to add a restriction to the spec that finalizers must not refer to their ForeignPtr (if they are ever to run) and instead pass it as an argument to the finalizer. Still much better than these awkward C land finalizers. Cheers, Manuel
I have to say that, given Simon's patch, I am inclined to revert back to the old API for foreign pointers.
I don't think such a change should be made unless Malcolm and I are able to implement it. I'm not yet convinced that Simon's patch is as easy or correct as it seems and will not be until it has been heavily tested and until I have a chance to look carefully at the consequences of the change elsewhere in the system. Also, Malcolm reported using a similar trick but that he couldn't get it to work reliably (i.e., it was ok if the finalizer did nothing but call out to C but not otherwise).
The restriction on pure C land finalizers *is* awkward, and as we have already seen implies further changes (ie, adding something like `finalizerFree').
We missed a small detail in specifying the change and fixed it when we went to implement it. This happens with most design changes and doesn't seem like evidence of awkwardness to me. -- Alastair Reid alastair@reid-consulting-uk.ltd.uk Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/
Alastair Reid <alastair@reid-consulting-uk.ltd.uk> wrote,
I have to say that, given Simon's patch, I am inclined to revert back to the old API for foreign pointers.
I don't think such a change should be made unless Malcolm and I are able to implement it.
I'm not yet convinced that Simon's patch is as easy or correct as it seems and will not be until it has been heavily tested and until I have a chance to look carefully at the consequences of the change elsewhere in the system.
Also, Malcolm reported using a similar trick but that he couldn't get it to work reliably (i.e., it was ok if the finalizer did nothing but call out to C but not otherwise).
I won't change anything until we have finished discussing the issue, but I got the impression that I probably rushed the last change. So, I wouldn't consider the current definition the default either.
The restriction on pure C land finalizers *is* awkward, and as we have already seen implies further changes (ie, adding something like `finalizerFree').
We missed a small detail in specifying the change and fixed it when we went to implement it. This happens with most design changes and doesn't seem like evidence of awkwardness to me.
Fair enough. However, I still think that the current definition is awkward. It is awkward as it goes against the spirit of the rest of the spec, which strives to accomplish as much as possible in Haskell, not in C land. It's awkward as a user with a fairly big application (namely George) is inconvenienced by it. It's awkward because it requires a subtle side condition on what is a valid finaliser that is going to trip up users and cause strange errors. This awkwardness is tolerable if it simplifies the implementation a lot (as in avoids having to implement pre-emptive concurrency). It is not tolerable if it is just a matter of verifying that an existing implementation proposal does indeed work (or even if it requires tweaking that proposal). At least, this is my understanding of good language design. Cheers, Manuel
It's awkward as a user with a fairly big application (namely George) is inconvenienced by it.
I remain unconvinced that this is an issue. George, can you describe what the entrypoints you require are in more detail?
It's awkward because it requires a subtle side condition on what is a valid finaliser that is going to trip up users and cause strange errors.
It is the same side condition we place on unsafe calls.
This awkwardness is tolerable if it simplifies the implementation a lot (as in avoids having to implement pre-emptive concurrency).
See my other posting this morning on why Haskell finalizers are of no practical use without adding concurrency primitives which support blocking.
It is not tolerable if it is just a matter of verifying that an existing implementation proposal does indeed work (or even if it requires tweaking that proposal).
Another valid goal is portability. We can specify a low bar now and raise the spec later. For example, at present the spec is that the finalizer is equivalent to an unsafe ccall, we can strengthen that spec to allow safe ccalls too later (and once we can do that we can extend the spec with convenience functions which allow naked Haskell finalizers). A
Alastair Reid <alastair@reid-consulting-uk.ltd.uk> wrote,
It's awkward because it requires a subtle side condition on what is a valid finaliser that is going to trip up users and cause strange errors.
It is the same side condition we place on unsafe calls.
Safe calls are the default. If somebody explicitly specifies an unsafe call, I expect the person knows what s/he is doing.
This awkwardness is tolerable if it simplifies the implementation a lot (as in avoids having to implement pre-emptive concurrency).
See my other posting this morning on why Haskell finalizers are of no practical use without adding concurrency primitives which support blocking.
This point is orthogonal to my motivation of wanting Haskell finalizers. (1) Haskell finalizers fit better into the rest of the system, where we do in Haskell whatever we can. (2) C finalzers have this awkward side condition on re-entering the Haskell system - see above. Moreover, if a system supports concurrency primitives and blocking, Haskell finalizers fit smoothly in. If we have C finalizers, we need to add Haskell finalizers in such a system and have two redundant mechanisms (otherwise, we force people to write a superflous C function re-entering Haskell for each Haskell finalizer they need). It's just yucky.
It is not tolerable if it is just a matter of verifying that an existing implementation proposal does indeed work (or even if it requires tweaking that proposal).
Another valid goal is portability.
We can specify a low bar now and raise the spec later. For example, at present the spec is that the finalizer is equivalent to an unsafe ccall, we can strengthen that spec to allow safe ccalls too later (and once we can do that we can extend the spec with convenience functions which allow naked Haskell finalizers).
To be honest, I don't understand why you are so opposed to this. Why not find out whether SimonM's patch works? If it does, the problem is solved, we can use the nicer definition, and everybody is happy. Manuel
To be honest, I don't understand why you are so opposed to this.
I don't want to encourage people to write code which can't work on Hugs and NHC. If we provide Haskell finalizers with the restriction that they can only call C code, users will go ahead and write finalizers which call arbitrary Haskell code and then the code won't work with Hugs and NHC - portability is decreased. So, maybe we should make Haskell finalizers work as well on NHC and Hugs as they do on GHC? What would it take to do this? I believe it would take a massive effort to make Haskell finalizers work both for existing systems and new systems because: 1) Being able to write finalizers in Haskell is pointless unless they can safely modify shared state. (Worse, it is dangerous because unsafely modifying shared state leads to race conditions which are unbelievably hard to track down. Much harder than the debugging sessions that George envisages.) Any useful form of Haskell finalizer requires the provision of blocking synchronization which will require a complete rewrite of the runtime system and potentially decrease portability. 2) None of our Haskell primitives are intended for a multi-threaded environment. Consider a pair of primitives that share a common data structure like hClose and open. What happens if a finalizer calls hClose while the main thread is in the middle of creating a file? It probably works ok but we know from experience with writing multithreaded code that the only way to be sure is to look at the code. In fact, we have to check it and every other Haskell primitive that affects some shared structure. In other words, we have to vet a lot of code to see if it can be used in a single-threaded way and modify offending code by moving any allocations to the start or end of the modifications. We also have to keep this in mind every time we modify the runtime system. In short, we have to do most of the work of making Hugs support preemptive concurrency: we have to provide blocking synchronization and we have to rewrite our code as though a preemption could happen on any heap allocation. Plus, it makes maintenance harder because it makes it easy to introduce race conditions.
Why not find out whether SimonM's patch works? If it does, the problem is solved, we can use the nicer definition, and everybody is happy.
As you know, testing for race conditions is very hard. We could test and test and not see a problem and then a week into the release we get a bug report. But, worse, the report also says that the bug goes away if they make some small change to their code. The change will not actually fix the problem, it will just change the allocation/GC pattern so that the problem doesn't show up in the test. -- Alastair Reid alastair@reid-consulting-uk.ltd.uk Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/
PS: Is everybody going to be at PLI'02? Then, we could discuss this face to face. Manuel
You do have to worry about the impact of running Haskell finalizers at any point during execution of Haskell code. Since you can't do any concurrency synchronisation between the finalizer and the main Haskell thread, shared access to mutable data is impossible. This is just something you'd have to document carefully (note, it's not something you can do with the current finalization mechanism either, but there it's obvious).
This is a pretty large restriction since the only reason for running Haskell finalizers is to manipulate shared Haskell state. Here's a typical example of where you would need Haskell finalizers instead of C finalizers: consider a simple graphics library which maintains a list of windows and deletes a window from the list when its finalizer runs (it presumably uses weak pointers to avoid the space leak). If the finalizer blindly deletes the object from the list, we get a race condition. If the object is protected by a lock, the finalizer can't access it. If we want to allow finalizers to access objects protected by a lock, we have to be prepared to block execution of the finalizer until the main thread releases the lock. I claim that Haskell finalizers are no use unless we provide synchronization primitives, those primitives are able to block execution of a currently running finalizer. I also claim that implementing blocking is a significant undertaking. Simon's patch doesn't make Hugs' concurrency implementation work with finalizers and it will take a fairly significant rewrite of the implementation to make it work. (At present, it doesn't work with unsafePerformIO.) I don't know if I'd be able to support blocking in any useful way without reimplementing the whole thing on top of a native thread library (pthreads, etc.) AFAIK, NHC does not implement any concurrency primitives so NHC would have to add them before it can provide any useful form of Haskell finalizers. -- Alastair Reid alastair@reid-consulting-uk.ltd.uk Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/
participants (3)
-
Alastair Reid -
Manuel M T Chakravarty -
Simon Marlow