John's approach is the right one, but note that the "STM setup work" will be visible to other transactions. Any work done there must be considered consistent as it will be committed. For instance if we have the following:
atomically $ do
m <- readTVar mutex
check (not m)
writeTVar m True
writeTVar a True
ffiCall
atomically $ do
writeTVar b True
writeTVar m False
If it should be the case that `a` is true only if `b` is true then we could run into problems with some other transaction that is not concerned with the `ffiCall` but is concerned about `a` and `b`.
Another approach that might be viable is to just fix your foreign call's interface by putting a lock in the foreign code. If it really is a pure call, then there is no danger of deadlocking there. This will then be as safe as any foreign call inside STM. There are dangers here of course. One particular danger is if your call takes multiple arguments that need to be consistent, you can see inconsistency inside a failed transaction before the runtime system has determined that it is a failed transaction. For instance:
atomically $ do
x <- readTVar a
y <- readTVar b
return $ unsafePerformIO $ ffiCall x y
Say `x` is an array and `y` is an index into that array. Even if your transactions keep these consistent, you could send inconsistent data to the foreign call. You can fix this by checking (at the value level) the integrity of the data or changing the granularity of the data and putting values `x` and `y` into the same `TVar`.
Beyond this problem I don't know what other issues foreign calls inside STM face, but I would love to hear what others know.
Ryan