September 2002 - FFI - Haskell.org

RE: Updates to FFI spec
by Simon Marlow 11 Sep '02

11 Sep '02

> >> - I'd like to see a standard way to call the GC from C > >> > >> http://www.mail-archive.com/ffi@haskell.org/msg00565.html > >> > >> Note that Hugs and GHC have had this for ages except that we call > >> the function 'performGC' and there's no way to control how many > >> generations are collected. > > > I don't have a strong opinion on this one. > > Maybe SimonM and Malcolm do. > > I know GHC has an interface already and I suspect that NHC will too. > GHC and Hugs both call it performGC but this doesn't match the naming > convention used in the ffi and it is expensive to use with > generational GC (since there is no way to indicate that a partial GC > may be enough). System.Mem.performGC does a major GC. When would a partial GC be enough? Isn't the property you really want "a full garbage collection has been performed and all finalizers have been run to completion"? Note this would be non-trivial to implement in GHC. We currently don't distinguish between finalizer threads and normal threads, preferring to leave it to the programmer to implement any synchronisation on finalizers using MVars. > >> - I see the question of Function prototypes as a portability > >> problem waiting to happen. Either Hugs and GHC are right (you > >> should use the user-supplied header file or NHC is right (you > >> should ignore the header file). They can't both be right if we > >> want portable code so the report should be clear about which one is > >> right. I think the spec should be clarified along these lines: Header files have no impact on the semantics of a foreign call, and whether an implementation uses the header file or not is implementation-defined. Some implementations may require a header file which supplies a correct prototype for the function in order to generate correct code. I'd be equally happy (perhaps happier) if the header file spec was removed altogether. In a sense, this would leave the Haskell part of a foreign binding even more portable, because it doesn't have to specify the names of header files which might change between platforms. There are already "external" parts of a binding such as the names of link libraries, I think I'm arguing that header files also fall into that category. > >> - Changes to hs_init > >> > >> http://www.mail-archive.com/ffi@haskell.org/msg00539.html > > > I am ok with that. Currently, there is a problem with the version > > that is in the spec and GHC in that GHC requires an extra argument > > to initialise modules. So, it all depends a bit on how far SimonM > > thinks its implementable. > > Hugs doesn't have anything resembling hs_init at the moment so I can't > say for sure. I think it can be done as a thin layer on top of the > the existing spec though so I don't think it's that hard. Perhaps on GHC you should be required to "register" the top module in your program first, maybe something like registerModule(__stginit_Main); that way you can register multiple modules (which isn't possible at the moment, you have to have another module which imports all the others). Cheers, Simon

5 7

Re: Proposed change to ForeignPtr
by George Russell 11 Sep '02

11 Sep '02

Malcolm Wallace wrote [snip] > Quite simply, no finaliser (whether in Foogle or Haskell) should > be capable of triggering a garbage collection within its call. > This condition is absolutely necessary to prevent a cascade effect > of cross-language garbage collections, where a finaliser in Haskell > could trigger a GC in Foogle which triggers another (nested) GC in > Haskell etc. [snip] Unfortunately some sort of cascade is exactly what we want and need when the Haskell finaliser indicates that Haskell is no longer interested in some Foogle object, which means Foogle can run a GC which indicates Foogle is no longer interested in some Haskell object and so on . . . > Thus, if Haskell.freeSomething calls Foogle.freeSomething, and > Foogle.freeSomething cannot cause a Foogle GC, then no Foogle > finalisers are run yet, and so Foogle *cannot* call the Haskell world > until the Haskell GC is complete. After that, it doesn't matter > when the Foogle finaliser decides to run. [snip] But surely Foogle has no way of knowing when the Haskell GC is over? Suppose Haskell does [enter GC] ... [run finaliser 1] ... [run finaliser 2] ... [leave GC] Then you want Foogle to delay any Haskell calls consequent on finaliser 1 until [leave GC], don't you? How can it? Is it really so difficult to create some queue of delayed functions which can be appended to from C and which nhc checks every time it does "leave GHC"?

2 2

Re: Proposed change to ForeignPtr
by George Russell 11 Sep '02

11 Sep '02

Malcolm Wallace wrote (about delayed finalizers): > This was the originally proposed solution for re-entrant finalisers > in nhc98 and Hugs. The problem is that, certainly for nhc98, it is > nigh impossible for the RTS to identify a "convenient" later moment > at which to run the delayed finaliser. Well, if the nhc98 RTS can't work out a time when it will be possible to run the delayed finalizer, how in creation do you expect the Foogle finalizer or whatever Foogle GC process it invokes to do so?

2 1

RE: Proposed change to ForeignPtr
by George Russell 11 Sep '02

11 Sep '02

Simon Marlow wrote > George: the problem with allowing callbacks from a finalizer is that in > order to implement this your system essentially has to be able to make a > pre-emptive context switch from the currently running Haskell > computation to the finalizer; this turned out to have a fairly high > implementation cost for nhc98 and Hugs, so we decided to amend the spec > instead. You'll still be able to do this in GHC, of course, but it > won't be portable. However in the case I cited it isn't particularly important to execute the callback at once. If calling Haskell at arbitrary times is unimplementable, could there instead be some way of some way like "wrapper" of exporting a Haskell function [args] -> IO () so that the resulting C function, when executed, merely registers the Haskell action to be executed as soon as the Haskell RTS finds it convenient, rather than at once?

2 2

Re: Proposed change to ForeignPtr
by George Russell 11 Sep '02

11 Sep '02

Malcolm Wallace wrote [snip] > I don't see the problem. The Foogle garbage collector runs separately > and asynchronously to the Haskell GC. A Foogle object is released > by the Haskell collector, then at a later moment in time, a Haskell > object is released by the Foogle collector. Where is the conflict? [snip] Does "at a later moment in time" mean that it is late enough that we can be sure calling Haskell will be OK? Look, suppose for simplicity that Foogle implements an identical FFI to Haskell. So we have (Haskell ForeignPtr A) == (Foogle StablePtr A) (Foogle StablePtr A) points to (Foogle ForeignPtr B) (Foogle ForeignPtr B) == (Haskell StablePtr B) Haskell frees (Haskell ForeignPtr A) thereby causing the finalizer. This presumably does the Foogle equivalent of freeStablePtr on A. This *may* trigger an immediate Foogle garbage collection (I think Foogle's RTS is within its rights if it does), so that Foogle now wants to finalize Foogle ForeignPtr B. The Foogle finalizer action for foreignPtr B involves calling back to Haskell so that (Haskell StablePtr B) can have freeStablePtr applied to it. Now is the FFI specification going to guarantee that however quickly the Foogle garbage collector executes the Haskell finalizer, things will work. It seems to me the wording suggested does not guarantee this. All it says that the finaliser cannot portably "call back into the Haskell system". But when the finaliser provokes an immediate GC which calls back into the Haskell system? I'm altogether rather puzzled by this notion of FunPtr's which are allowed to call Haskell back at some times and not others. Nor do I understand how it's supposed to work, say, on truly parallel Haskell implementations. [snip] > Why do you suggest a need for reference counts? In the absence > of cycles, surely the existing two garbage collectors (howsoever > implemented) are sufficient. [snip] Garbage collectors need roots. If I understand the situation correctly, a StablePtr is itself a root, until explicitly freed. If you are handing out the same object via the same StablePtr to several different Foogle things, it seems to me you might attach a reference counter to the StablePtr, so that when Foogle says "I am no longer interested in this (Haskell) StablePtr", you decrement the reference counter by 1, and deallocate if it reaches zero. Alternatively of course you could create a fresh StablePtr every time something is passed to Foogle. I think reference counters are sometimes used in things like CORBA, where a similar problem arises. Of course they cannot cope with interlanguage cycles.

1 0

RE: Proposed change to ForeignPtr
by Simon Marlow 10 Sep '02

10 Sep '02

Malcolm says: > I don't see the problem. The Foogle garbage collector runs separately > and asynchronously to the Haskell GC. A Foogle object is released > by the Haskell collector, then at a later moment in time, a Haskell > object is released by the Foogle collector. Where is the conflict? Foogle might be doing reference counting instead of GC, so it might want to run the finalizer synchronously. George: the problem with allowing callbacks from a finalizer is that in order to implement this your system essentially has to be able to make a pre-emptive context switch from the currently running Haskell computation to the finalizer; this turned out to have a fairly high implementation cost for nhc98 and Hugs, so we decided to amend the spec instead. You'll still be able to do this in GHC, of course, but it won't be portable. Cheers, Simon

1 0

Proposed change to ForeignPtr
by Alastair Reid 10 Sep '02

10 Sep '02

Since requiring ForeignPtr.newForeignPtr would require preemptive concurrency (see previosu discussion), I propose the following changes: 1) Add these functions: makeForeignPtr :: Ptr a -> FunPtr (Ptr a -> IO ()) -> IO ForeignObj attachForeignPtrFinalizer :: ForeignPtr a -> FunPtr (Ptr a -> IO ()) -> IO () 2) Remove newForeignPtr and addForeignPtrFinalizer Two variants on this proposal: 1) Reuse the old function names instead of inventing new ones. I don't care much either way. 2) Instead of removing the old functions, rename them and mark them as an optional extra. I think this would be a mistake - the GHC documentation is the right place to mention extensions which are not available on all ffi implementations. I have implemented makeForeignPtr in Hugs (I didn't do attachForeignPtrFinalizer yet because it'll take a bit longer to add that functionality and we can compile most (all?) of the hierarchial libs without it. -- Alastair Reid alastair(a)reid-consulting-uk.ltd.uk Reid Consulting (UK) Limited http://www.reid-consulting-uk.ltd.uk/alastair/

3 8

RE: Updates to FFI spec: hs_init() & friends
by Simon Marlow 10 Sep '02

10 Sep '02

> So, my proposal is to > > * permit multiple calls to hs_init() and hs_exit() (the > number of calls must match of course), > > * command line arguments specified in the second and > following invocations to hs_init() are ignored, and > > * both arguments to hs_init() may be NULL (to indicate the > absence of arguments). > > The second point is necessary to keep the implementation on > the Haskell side reasonably straight forward. > > I think, we also need to say something about the timing of > hs_set_argv() and calling getProgName and getArgs. I'd say, > if hs_set_argv() is used at all, it must be before the first > call to getProgName and getArgs (otherwise, the system > behaviour is undefined). > > Objections? Fine by me. > > Perhaps on GHC you should be required to "register" the top > module in > > your program first, maybe something like > > > > registerModule(__stginit_Main); > > > > that way you can register multiple modules (which isn't > possible at the > > moment, you have to have another module which imports all > the others). > > So, registerModule() would be a GHC-specific extension to > the FFI API? Well, more of a restriction than an extension :-) But yes, that would be GHC-only. Cheers, Simon

1 0

RE: Updates to FFI spec: performGC
by Simon Marlow 10 Sep '02

10 Sep '02

> > > Hmmm, the garbage collector is a black box and has its own > > > complicated heuristics for managing memory usage, but you are > > > describing a mechanism that depends rather heavily on certain > > > assumed behaviours. At the least, that gives the garbage > collector > > > less flexibility to change its own behaviour, lest it > invalidate the > > > assumptions made by the external allocator. > > > > This is a situation where something is better than nothing. The > > heuristics we used with Hugs were pretty lame -I don't > think they even > > worked as I'd intended- but they were good enough to get > the job done. > > In fact, the application ran brilliantly: small GC pauses, little > > wasted memory. > > I am not sure what to do with this one. SimonM, did > Alastair convince you by now that what he proposes is > useful? Well, perhaps, but I objected on the grounds that it couldn't be done portably. Within the bounds of a single system, having such functionality could well be useful, but it would be dependent on the characteristics of the memory manager of that particular system. I'd leave such things to individual implementations. Cheers, Simon

1 0

RE: module Data.Bits
by Simon Marlow 09 Sep '02

09 Sep '02

> I have just been implementing the library module `Data.Bits' > for nhc98, > as part of an effort to bring that compiler up-to-date with > the latest FFI spec. > > The FFI spec says that the operations called 'shift' and 'rotate', > shift and rotate their argument to the right. However, the GHC > implementation in CVS shifts and rotates to the left, and is > documented to do so. > > Who is right? FWIW, before I read any documentation I expected the > undecorated forms to be rightward. I think it's probably a mistake in the FFI addendum. As far as I can tell, the GHC/Hugs implementation of Bits has always worked the other way around. Cheers, Simon

1 0