October 2002 - FFI - Haskell.org

RE: The Revenge of Finalizers
by Simon Marlow 18 Oct '02

18 Oct '02

> Simon Marlow wrote: > [snip] > > Don't you run into a problem even if the two threads use the same > > ordering? Suppose > > > > - thread 1 does the atomicModifyIORef, and gets preempted before > > doing the seq > > - thread 2 does its own atomicModifyIORef, and the seq. Thread 2 > > gets an inconsistent view of the IORefs. > > No. At the time of the seq in thread 2, the thunk (call it Thunk 2) > returned by the second atomicModifyIORef will refer to the > thunk (Thunk 1) > put in there by the first atomicModifyIORef. The seq will cause the > unsafePerformIO from the second atomicModifyIORef to be > evaluated. This > will come to the "if" statement which will demand the value > of Thunk 1. > This will in turn provoke the evaluation of Thunk1. This will cause > the action of thread 1's simpleToggle2 to be evaluated to completion. Ok, so it relies crucially on the fact that the 'if' causes evaluation. (you might have written it to be more lazy, so that the pair was returned immediately without evaluating contents1 first, for example). In that case, yes I agree it is horrible :) Cheers, Simon

3 4

Objections to runAtomically
by George Russell 18 Oct '02

18 Oct '02

I don't like runAtomically either, because once again it assumes a global lock. This is fine for GHC or Hugs or NHC on single processors, but it would be a pain if you had multiple processors. You can't avoid locking between processors altogether of course; atomicModifyPVars and the unsafePerformIO hack I just posted require data to be updated consistently (you don't want the same unsafePerformIO being evaluated twice). However that is not so bad because the locking is only on particular bits of data; it's reasonable that access to particular thunks or PVars will be "owned" by one processor or another, so that if two processors are working on independent bits of state they can chug along perfectly happily without locking each other, and only need to talk when one processor wants to get at a bit of state owned by another. Although Alastair suggests compilers might try to spot when runAtomically can be used safely without stopping all but one processor, I think analysis of IO actions to check for potential conflicts might be even harder than getting Hugs to run Haskell finalizers 8-) It would require region analysis and probably worse. However we don't really need to discuss this anyway, since I don't think either runAtomically or atomicallyModifyIORef need to be in the FFI standard. I'm quite happy to leave this open.

2 10

RE: The Revenge of Finalizers
by Simon Marlow 18 Oct '02

18 Oct '02

> > However in general I think we can hide some of the horribleness from > > the user: > > > modify2IORefs :: IORef a -> IORef b -> (a -> b -> (a,b,c)) -> IO c > > [horrible code deleted] > > And if they need to update 3 IORefs or a list of IORefs? > > Writing code like that yourself and getting it right and portable > between compilers seems to be ludicrously hard. > > I can't tell if that code is right (my gut says no). Worse though, I > don't even know what semantic framework to use to reason about it if > we want to be sure the code will work in the presence of strictness > analyzers, eager evaluation, parallel evaluation, fully-lazy > evaluation, etc. Operational reasoning and reasoning by example > struggle with such a task. I agree. However, I think we're trying to solve a problem that doesn't exist yet. All the libraries we have which are affected can be fixed by using atomicModifyIORef, and even if one were to arrive which can't be fixed in this way, the chances that someone would also want to use it from a finalizer are rather low. I think a more pressing problem is whether Haskell finalizers can be implemented in Hugs properly - any thoughts on the problem I raised earlier? Cheers, Simon PS. I've implemented atomicModifyIORef in GHC and am testing it now.

2 1

RE: The Revenge of Finalizers
by Simon Marlow 17 Oct '02

17 Oct '02

> simpleToggle2 :: IORef Bool -> IORef Bool -> IO (Just (Bool,Bool)) > > which attempts to flip the two IORefs from True to False (if > they are both True); > otherwise returning their actual values. Then you could code > this something like this > (no I'm not going to check if it passes GHC) > > simpleToggle2 ioRef1 ioRef2 = > do > res <- atomicModifyIORef ioRef1 (\ contents1 -> > unsafePerformIO (atomicModifyIORef ioRef2 (\ contents2 -> > if contents1 && contents2 > then > (False,(False,Nothing)) > else > (contents2,(contents1,Just (contents1,contents2))) > ) > ) > seq res (return res) > > Then the first atomicModifyIORef replaces the contents of > ioRef1 by a thunk, and > returns another thunk. The seq then proceeds to evaluate > this thunk, causing > the unsafePerformIO to be run, which changes ioRef2. > > There IS a potential problem, because if you run > simpleToggle2 on the same ioRef > > simpleToggle2 ioRef ioRef > > or if two threads run simpleToggle2 simultaneously on the > same ioRefs but in opposite order > > simpleToggle2 ioRef1 ioRef2 || simpleToggle2 ioRef2 ioRef1 Don't you run into a problem even if the two threads use the same ordering? Suppose - thread 1 does the atomicModifyIORef, and gets preempted before doing the seq - thread 2 does its own atomicModifyIORef, and the seq. Thread 2 gets an inconsistent view of the IORefs. right? Simon

2 1

RE: The Revenge of Finalizers
by Simon Marlow 17 Oct '02

17 Oct '02

> > I don't know how to achieve the same goal with > > atomicModifyIORef. > > I do. To modify ioRef1 and ioRef2 "simultaneously", write > > atomicModifyIORef ioRef1 (\ contents1 -> unsafePerformIO > ioRef2 (\ contents2 -> blah blah)) > > The actual modification will take place when the result or > contents of ioRef1 or ioRef2 get evaluated. I don't understand how this works. Unless I'm missing something, you'll see the contents of ioRef1 at the point at which the first atomicModifyIORef takes place, but the contents of ioRef2 from the time at which the unsafePerformIO is evaluated, which is some unspecified time later. Also another modification might come along before the first unsafePerformIO is evaluated, and it might see the "new" value of ioRef1 but the old value of ioRef2, if it doesn't evaluate the value of ioRef1 early enough. Nevertheless, I believe that the addition of atomicModifyIORef will let us write System.Random and Data.Dynamic in a safe way - both libraries have only a single global IORef to worry about. Cheers, Simon

2 1

RE: The Revenge of Finalizers
by Simon Marlow 17 Oct '02

17 Oct '02

> > So is it just that there are a small number of places where it is > > unsafe to invoke IO from a primitive, > > It is unsafe to invoke IO from a prim _in the middle of_ manipulating > a data structure manipulated by IO operations unless the data > structure is designed to allow that nested use. Trivial example: > using a counter instead of a bool to record whether finalizers are > blocked makes nested calls ok (at least as far as that particular data > structure is concerned). > > (It would be unsafe to invoke eval from prims which manipulate data > structures in pure code except that we took care when writing the code > to allow for that since most of the prims invoke eval.) > > > [...] Can we easily identify which are the unsafe places > and fix them? > > Look at all prims with types in IO. Look at what data structures they > touch. Check if there are accesses to that data structure 'both > sides' of a call to eval (there may be a few functions which invoke > eval so watch for them). Check if any such accesses break data > structure invariants. Since this can affect Hugs as it stands (when you use unsafePerformIO), would you mind taking a look? > Now repeat for library code that we ship with Hugs. Data structures > of interest will be those constructed with IORefs and things built > from them (this includes MVars since Hugs implements MVars using > IORefs). The following libraries in fptools/libraries use IORefs and might be unsafe: System.Random and Data.Dynamic. Hugs has its own versions of these libraries that also use IORef in the same way. For GHC, we really ought to fix these to use MVars, since they're unsafe already. For Hugs, I don't think it would be a complete disaster if in the errata you say that Data.Dynamic and System.Random can't be used from finalizers. Cheers, Simon

3 3

RE: declaring C enum types
by Simon Marlow 17 Oct '02

17 Oct '02

> > > - If not, then how should enum values be declared in the FFI? > > > > What you need to do is run a little autoconf-like program which > > constructs a program containing a suitable example, runs it > through a > > C compiler and tells you what's going on. > > > > hsc comes very close but I'm not certain if it does exactly what you > > need. If not, compiling and running this program should tell you: > > Hmm, careful. The C compiler is free to be clever, and use a > char if there are <=256 elements in the enum, and short or > int otherwise. You want to know the size of your particular > enum, not any random enum. Yes. The C99 spec says (sec. 6.7.2.1): [#4] Each enumerated type shall be compatible with an integer type. The choice of type is implementation-defined but shall be capable of representing the values of all the members of the enumeration. [...] If a C compiler can choose any integer type, then it seems to me that mixing code from two C compilers on the same platform might not work. Hmmm. Anway, I think Alastair meant to say hsc2hs (not hsc), but you do actually get the right answer doing it this way. In hsc2hs if you say (#type enum foo) you'll get a Haskell type which corresponds to the actual type used by the C compiler (provided hsc2hs is using the same C compiler as was used to compile the library, presumably). Cheers, Simon

2 1

declaring C enum types
by Antony Courtney 17 Oct '02

17 Oct '02

Hi all, A relatively mundane question: Is there a preferred means of writing a foreign import declaration for a C function with a prototype that takes an "enum" argument? i.e., in C we have: enum bar { GOOD, BAD }; void foo(enum bar x); I recall from some sessions with gdb some years ago that most C compilers encode enums as if they were unsigned short values (i.e. 16 bits on most platforms). Does anyone know off hand: - Is it a safe, portable assumption that enums will always be encoded as an unsigned short or 16-bit word? - If so, should one prefer "Word16" or "UShort" on the Haskell side for the foreign import declaration? - If not, then how should enum values be declared in the FFI? Appologies if the FFI addresses this issue and I have overlooked it. Thanks, -antony -- Antony Courtney Grad. Student, Dept. of Computer Science, Yale University antony(a)apocalypse.org http://www.apocalypse.org/pub/u/antony

7 7

The Revenge of Finalizers
by George Russell 17 Oct '02

17 Oct '02

Malcolm Wallace wrote [snip] > I don't think MVars make any sense without concurrency, so I don't see > the point of implementing the one without the other. I'm willing to > be persuaded otherwise. (It would be nice to have some concurrency > in nhc98, of course, but I don't foresee that happening soon.) > > > I apologise to Malcolm, but in any case I doubt if implementing > > MVars in a world with only one thread would cost him a great deal of > > effort. > > Well it would be simple enough provided you didn't mind your > computation halting with deadlock rather unpredictably. :-) [snip] OK, the point is to make programs more portable from NHC to other systems. (Perhaps you would rather lock them in? 8-) If I write, say, Alastair Reid's newObject and killObject in NHC using IORefs, then this will work fine in NHC, but may mysteriously go wrong when someone ports them to GHC and calls newObject and killObject simultaneously. I'm not sure we should be discussing this here; really we should have a separate Haskell Mutable State standard. However let's try anyway. Clearly it would be trivial to implement MVars in NHC to fit such a goal, something like newtype MVar a = MVar (IORef (Maybe a)) takeMVar (MVar ioRef) = do aOpt <- readIORef ioRef case aOpt of Nothing -> error "Computer deadlocked. Please buy a new one." Just a -> do writeIORef ioRef Nothing return a (newMVar and putMVar I'll leave as exercises for the reader.) I admit though that Alastair is right in this: while NHC + MVars and NHC + Haskell finalizers may be doable, NHC + Haskell finalizers + MVars may not be, because then you have the problem of blocking. If the main program and the finalizer both do x <- takeMVar mVar putMVar mVar (update x) then the program will unnecessarily deadlock if the finalizer is called inbetween the take and the put. Malcolm, is it feasible to implement a workaround to this? However even if Haskell finalizers + MVars are impossible in NHC, I don't think Haskell finalizers + mutable state have to be. For example another mutable variable we could have would be a PVar which is always full and has functions newPVar :: a -> IO (PVar a) updatePVar :: PVar a -> (a -> a) -> IO a updatePVar represents an atomic update; the function is applied to the value in the PVar and the old value is returned. For GHC and Hugs you could implement PVars using MVars. NHC could implement them as follows: newtype PVar = PVar (IORef a) updatePVar (PVar ioRef) updateFn = do [stop any new finalizers running] a <- readIORef ioRef writeIORef (updateFn a) [reenable finalizers] return a Then we have mutable state (enough to implement Alastair's killObject/newObject safely) in a way that I think NHC wouldn't find it too hard to implement. Of course you need a global flag which disables removing things from the finalizer queue, and the reenable-finalizers primitive could also check to see if any finalizers had arisen in the meantime. Also we are implicitly assuming that the execution of finalizers is properly nested, but that doesn't seem to hard a condition to ensure (in the context of NHC). Malcolm, does all this make sense to you, or am I talking total rubbish? I don't claim these PVars are perfect; there are things I can do with MVars which I don't know how to do with PVars. But I do claim that they provide sufficient mutable state to be useful in finalizers, without being hard for anyone (GHC, Hugs or NHC) to implement with finalizers. A "Mutable Variables Standard" could then define IORefs and PVars. Then with Haskell98 + FFI + Mutable Variables you would be able to write portable code that does mutable things in finalizers, and we would all be happy.

4 11

RE: Why I want Haskell finalizers
by Simon Marlow 17 Oct '02

17 Oct '02

> > No, that's only a partial (and indeed very incomplete) solution. It > > relies on the Java GC knowing that that particular reference to the > > Haskell StablePtr is the only one that matters, and vice-versa for > > the Haskell GC. > > So you want StablePtrs to contain a reference count, a new function > to increment the count and for freeStablePtr to decrement the count? > > We toyed with designs like that but always found other ways to achieve > the same goal - but it's easy enough to do. > > A variant is for newStablePtr to look for an existing StablePtr to the > same thing. We avoided this design because of the semantic issues in > testing Haskell pointers for equality but mostly to avoid imposing a > lookup cost that we'd rarely benefit from. GHC's StablePtr implementation does all this, BTW. We were considering removing it because of the overhead, though. Cheers, Simon

1 0