Hello, Judging by the silence that greeted my last posts re this a couple of weeks ago I suspect this is of no interest to anyone but me :-( An extra couple of weeks cogitation on this hasn't really changed MHO much (other than I'm no longer so pessimistic about the safety of my <+ operator). Seeing as the FFI seems to be close to finalization I thought I'd have another go. So excuse me if this seems half baked, but am I the only one who thinks the type signatures of utilities like this.. allocaBytes :: Int -> (Ptr a -> IO b) -> IO b ..are rather inconvenient? Wouldn't.. allocaBytes :: Int -> (Ptr a -> b) -> b ..be more useful? To get around this I defined an operator.. infixl 0 <+ (<+) :: (Ptr a -> b) -> ((Ptr a -> IO b) -> IO b) -> b f <+ alloc = unsafePerformIO $ alloc $ \p -> return $ f p (Actually I now think the type of <+ should be.. (<+) :: (a -> b) -> ((a -> IO b) -> IO b) -> b ..so it will work with functions like withCStringLen) This seems to work, but I wasn't entirely comfortable with its safety due to the use of unsafePerformIO. Anyway, perhaps there's a good reason for things having been done this way, but right now I can't see it. So could somebody who understands the reasons for the FFI design decisions and underlying implementation explain why a change such as I have suggested could not or should not be made? Possible explanations that occur to me are.. 1- It's technically impossible. 2- It's unnecessary because FFI users can define their own contraptions like <+ if they want them, assuming the use of unsafePerformIO in this context really is safe (and unsafePerformIO is available). 3- It's undesirable because the existing forms are better in some way. 4- It's undesirable because it would break existing code (not true AFAICS). Thanks -- Adrian Hey
So excuse me if this seems half baked, but am I the only one who thinks the type signatures of utilities like this.. allocaBytes :: Int -> (Ptr a -> IO b) -> IO b ..are rather inconvenient?
Wouldn't.. allocaBytes :: Int -> (Ptr a -> b) -> b ..be more useful?
Why? What could the pure function you're passing to allocaBytes do with the pointer? When would the storage be freed again? The original allocaBytes frees it after the IO action has finished, but before returing itself. How would that apply to the pure version?
To get around this I defined an operator.. infixl 0 <+ (<+) :: (Ptr a -> b) -> ((Ptr a -> IO b) -> IO b) -> b f <+ alloc = unsafePerformIO $ alloc $ \p -> return $ f p
What can f do with the pointer, outside the IO monad, without using unsafePerformIO itself? Also, if you're passing alloca or allocaBytes for the alloc parameter, how do you make sure that 'f' has been _completely_ evaluated before alloca deallocates the storage again?
Possible explanations that occur to me are.. 1- It's technically impossible. 2- It's unnecessary because FFI users can define their own contraptions like <+ if they want them, assuming the use of unsafePerformIO in this context really is safe (and unsafePerformIO is available). 3- It's undesirable because the existing forms are better in some way. 4- It's undesirable because it would break existing code (not true AFAICS).
5- because this use of unsafePerformIO is highly dangerous (I think it will lead to segfaults sooner or later) and should be avoided at all costs. Also, 6- because a pure function cannot do anything useful with a pointer, IMHO. Cheers, Wolfgang
Hello Wolfgang, Thanks for your reply, you highlight the issues I've been trying to answer myself, though I don't think I came to the same conclusions. On Wednesday 28 May 2003 10:51, Wolfgang Thaller wrote:
So excuse me if this seems half baked, but am I the only one who thinks the type signatures of utilities like this.. allocaBytes :: Int -> (Ptr a -> IO b) -> IO b ..are rather inconvenient?
Wouldn't.. allocaBytes :: Int -> (Ptr a -> b) -> b ..be more useful?
Why?
I think it would be nice to free users from this style of programming.. withCString s $ \sPtr -> allocaBytes n $ \buffPtr -> withForeignPtr f $ \fPtr -> do ... It would be nice to just partially apply a foreign function of type.. f :: CString -> CString -> IO CInt .. to a Haskell string, as in.. g :: CString -> IO CInt g = f <+ withCString "Hello" Of couse there's nothing to stop you partially applying the function that results from the explicit lambdas code above, but only at the expense of re-marshalling the same Haskell string every time the underlying foreign function is called. Another example, this foreign import seems quite legal (not necessarily safe, but legal:-) foreign import ccall unsafe "cfunc" cfunc :: CString -> Word32 but its awkward to use it in conjunction with withCString. withCString "Hello" cfunc gives a type error (of course), you need this instead.. (unsafePerformIO $ withCString "Hello" (\p -> return $! cfunc p))
What could the pure function you're passing to allocaBytes do with the pointer?
[My understanding of Haskell was that all functions are pure, even if their type is (Ptr a -> IO b). What's impure is the action itself (which always has type IO <something>).] But who says the function is pure anyway, I never did:-) It just has some type other than (Ptr a -> IO b), such as (Ptr a -> Word32 -> IO b). To me, the type sigs of functions like withCString look like an (IMHO misguided) attempt to use the type system to enforce safe useage, at the expense of convenience for whoever it writing the binding. The reason I think it's misguided is that pretty much all safety with foreign interfaces is a matter of trust (not enforcable by the Haskell type system). Now it might well be that with current implementations it really is unsafe to use withCString with any function that is not of type (CString -> IO b). What I want to know is.. * is that so? (maybe) * should that be so? (no IMHO) What I want is to allow functions which take more than 1 argument to be used as the second argument of allocaBytes, withCString.. Typically the type of these would be.. Ptr a -> ... -> IO z though I'm not sure there's any good semantic reason to constrain them this way. I don't see any significant difference between proper treatment of pointers and proper treatment of IORefs. For example functions of type.. IORef a -> [IORef a] -> [IORef a] are perfectly ok, despite the fact that they can't do anything (I.E read or write) with their IORefs. That said, there might well be implementation difficulties that I'm not aware of.
When would the storage be freed again? The original allocaBytes frees it after the IO action has finished, but before returing itself. How would that apply to the pure version?
Without understanding the technical details of various Haskell implementations I can't say for sure, though I would imagine the storage would be freed when the corresponding pointer became garbage. This assumes the storage space is (haskell) heap allocated (as I gather it is in ghc from Simon M's reply). If it was stack allocated then I guess there would be problems.
Also, if you're passing alloca or allocaBytes for the alloc parameter, how do you make sure that 'f' has been _completely_ evaluated before alloca deallocates the storage again?
That was one of the problems I anticipated in my original posts, (danger of premature deallocation). I don't know the answer, that's why I asked.
Possible explanations that occur to me are.. 5- because this use of unsafePerformIO is highly dangerous (I think it will lead to segfaults sooner or later) and should be avoided at all costs.
You might well be right regarding my <+ operator, though I haven't seen any segfaults using this with ghc. I did test it by using this with a library binding I'm working on. I've subsequently removed all use of <+ because I couldn't figure out for sure whether it's safe or not. But if functions like withCString had the type sigs I'm asking for, it wouldn't be necessary to use unsafePerformIO in the first place. Regards -- Adrian Hey
Hello Adrian,
It would be nice to just partially apply a foreign function of type.. f :: CString -> CString -> IO CInt .. to a Haskell string, as in.. g :: CString -> IO CInt g = f <+ withCString "Hello"
OK... This is the special case of _constant_ values... It might be safe to write the following: newtype GarbageCollectedCString = GarbageCollectedCString (ForeignPtr CChar) toCString :: String -> GarbageCollectedCString After creation, the C String should never be modified (treat it as a const char* in C, not a char*). it would be allocated at some point in the future by the garbage collector. The with* and alloca* functions, however, return 'Ptr's, not 'ForeignPtr's. Ptrs are just address values, it's not possible to garbage-collect them or to attach finalizers (that's what ForeignPtrs are for, there's a slight performance hit). Also, as soon as you plan to modify the block pointed to by a ForeignPtr, you have to be in the IO monad or lose referential transparency.
Of couse there's nothing to stop you partially applying the function that results from the explicit lambdas code above, but only at the expense of re-marshalling the same Haskell string every time the underlying foreign function is called.
Another example, this foreign import seems quite legal (not necessarily safe, but legal:-) foreign import ccall unsafe "cfunc" cfunc :: CString -> Word32 but its awkward to use it in conjunction with withCString.
withCString "Hello" cfunc gives a type error (of course), you need this instead.. (unsafePerformIO $ withCString "Hello" (\p -> return $! cfunc p))
OK, that particular unsafePerformIO looks safe to me. However, if you have foreign import ccall unsafe "cfunc" cfunc :: CString -> Word32 -> Word32 (unsafePerformIO $ withCString "Hello" (\p -> return $! cfunc p)) 42 ... you're in terrible danger; Before applying the argument '42', the unsafePerformIO is evaluated; and before the argument '42' is applied, the unsafePerformIO will have to have _finished_ evaluating. That means withCString will be finished, which, in turn, means that withCString will already have deallocated the buffer for the string. Bad. Very bad. Perhaps it won't segfault, because the memory for the buffer won't have been reused and overwritten yet, but it's still bad.
What could the pure function you're passing to allocaBytes do with the pointer?
[My understanding of Haskell was that all functions are pure, even if their type is (Ptr a -> IO b). What's impure is the action itself (which always has type IO <something>).]
Sorry. Sloppy terminology on my part. My point was that allocaBytes (in it's current form) relies on getting it's hands onto the action. The action allocaBytes 42 f will first allocate 42 bytes, then execute the action (f ptr) and then free the 42 bytes again. If (f ptr) is not of type IO a, but of type a -> b (where b might be IO c or c -> d or ...), what can possibly happen between before the block is freed again?
When would the storage be freed again? The original allocaBytes frees it after the IO action has finished, but before returing itself. How would that apply to the pure version?
Without understanding the technical details of various Haskell implementations I can't say for sure, though I would imagine the storage would be freed when the corresponding pointer became garbage. This assumes the storage space is (haskell) heap allocated (as I gather it is in ghc from Simon M's reply). If it was stack allocated then I guess there would be problems.
It's a Ptr, not a ForeignPtr, so the Garbage Collector cannot keep track of whether it's still live or not. GHC's implementation of allocaBytes does some black magic for superior performance, and I think that's what SimonM was referring to. But the use of the Haskell heap is just an implementation detail here, not a feature... From the library source code (libraries/base/Foreign/Marshal/Alloc.hs): #ifdef __GLASGOW_HASKELL__ -- ... some completely unintelligible black magic ... #else allocaBytes :: Int -> (Ptr a -> IO b) -> IO b allocaBytes size = bracket (mallocBytes size) free #endif Look at the "generic" non-glasgow-haskell implementation: it just mallocs the block and then frees it, no chance for GC (malloc & free are really implemented using the C stdlib).
Now it might well be that with current implementations it really is unsafe to use withCString with any function that is not of type (CString -> IO b). What I want to know is.. * is that so? (maybe)
If you want to use Ptrs (and CString is basically Ptr CChar), yes, because the block has to be explicitly freed by the *action* withCString. If you use ForeignPtrs instead of Ptrs, you can have that, or something like the toCString function whose signature I presented above. But you can have that only if you "promise" not to modify the data block. For a function like allocaBytes, you will definitely want to modify the allocated block, so it's allocation and use has to be in the IO monad in order to be safe.
* should that be so? (no IMHO)
Yes, it should be. If I have a Ptr, I want to be able to do pointer arithmetic with it, and I want it to be represented as efficiently as possible. That means I have to stay in the IO monad. If I have a ForeignPtr, I don't want pointer arithmetic, and I want it to be garbage collected. Now the question we should be asking is, why isn't there a version of CString based on ForeignPtrs? Cheers, Wolfgang
Adrian Hey <ahey@iee.org> writes:
So excuse me if this seems half baked, but am I the only one who thinks the type signatures of utilities like this.. allocaBytes :: Int -> (Ptr a -> IO b) -> IO b ..are rather inconvenient?
I don't think the types are really as inconvenient as you suggest. To go back to your original posting a couple of weeks ago...
I think it would be nice to free users from this style of programming.. g s n f = withCString s $ \sPtr -> allocaBytes n $ \buffPtr -> withForeignPtr f $ \fPtr -> do g' sPtr buffPtr fPtr
g' :: CString -> Ptr a -> Ptr b -> IO c
... you suggested that what you would really like to write instead is just g :: String -> Int -> ForeignPtr a -> IO c g s n f = g' <+ withCString s <+ allocaBytes n <+ withForeignPtr f In order to make this work with your definition of <+, you needed to change the types. But perhaps that was because your definition of <+ was not exactly what you needed. How about the following alternative: g s n f = g' <$ withCString s <+ allocaBytes n <+ withForeignPtr f infixl 3 <$ a <$ b = b a infixr 4 <+ f <+ g = \h-> f $ \a-> g $ \b-> h a b which preserves all the types exactly as they are in the FFI standard? This avoids the whole question of introducing unsafePerformIO inside the <+ combinator, thus also avoiding its unsafety. Should you happen to want to marshall this way in the middle of pure non-IO code, then you can wrap a single unsafePerformIO at the outer level, e.g. g :: String -> Int -> ForeignPtr a -> c g s n f = unsafePerformIO $ do g' <$ withCString s <+ allocaBytes n <+ withForeignPtr f which is much more likely to preserve safety than having multiple embedded calls. Regards, Malcolm
On Thursday 29 May 2003 13:56, Malcolm Wallace wrote:
In order to make this work with your definition of <+, you needed to change the types. But perhaps that was because your definition of <+ was not exactly what you needed.
Nope. <+ as defined is exactly what I need, the only problem with it is it seems to be broken :-)
How about the following alternative:
g s n f = g' <$ withCString s <+ allocaBytes n <+ withForeignPtr f
infixl 3 <$ a <$ b = b a
infixr 4 <+ f <+ g = \h-> f $ \a-> g $ \b-> h a b
which preserves all the types exactly as they are in the FFI standard?
That's very neat, though I think a type sig would help users (well me:-) figure out what to do with this definition of <+ Here's the most appropriate for our purposes I think (not most general).. (<+) :: ((a -> IO z) -> IO z) -> (( b -> c) -> IO z) -> ((a -> b -> c) -> IO z) I don't think this is quite what I wanted, it seems hard to do the "partial application" trick with this. But even disregarding the issues regarding allocation lifetime, the partial application trick seems unsafe outside the IO monad because of the potential mutability of things that are pointed to. So maybe that's not really a valid gripe re. your definition of <+. To be honest I still find the whole idea of things that are pointed to disappearing at the most awkward times quite unsettling. I guess for the purposes of FFI programming I need to forget my "bad" FP habits and start thinking like C programmer :-) Regards -- Adrian Hey
On Thursday 29 May 2003 11:18, Adrian Hey wrote:
I don't see any significant difference between proper treatment of pointers and proper treatment of IORefs. For example functions of type.. IORef a -> [IORef a] -> [IORef a] are perfectly ok, despite the fact that they can't do anything (I.E read or write) with their IORefs.
Whoops, now I see a real problem with what I'm asking for. I guess since the (hypothetical) withCString is creating a new CString (bit like an IORef) it should have type withCString :: String -> (CString -> b) -> IO b and I guess my <+ should be.. (<+) :: (a -> b) -> ((a -> IO b) -> IO b) -> IO b f <+ alloc = alloc $ \p -> return $ f p (Well at least that's got rid of the unsafePerformIO :-) Would this be safe? Regards -- Adrian Hey
Whoops, now I see a real problem with what I'm asking for.
I guess since the (hypothetical) withCString is creating a new CString (bit like an IORef) it should have type
withCString :: String -> (CString -> b) -> IO b
It's still highly dangerous. When will the string be freed (no you can't rely on the garbage collector, because a CString is a Ptr and not a ForeignPtr; so the string has to be freed explicitly). Assuming your type for withCString... foreign import ccall foo :: CString -> Word32 -> Word32 main = do bar <- (withCString "Hello, world." foo) :: IO (Word32 -> Word32) print (bar 42) OK, what does this do: The first statement in "main": *) Allocates a buffer for the string and pokes the string in there. *) Binds the first parameter to foo (i.e., allocates a partial application (PAP) on the haskell heap). *) Frees the string buffer. The print statement: *) calls the foreign function "foo" with the pointer to the already-freed (!) string buffer and the number 42 as arguments *) prints the result
and I guess my <+ should be.. (<+) :: (a -> b) -> ((a -> IO b) -> IO b) -> IO b f <+ alloc = alloc $ \p -> return $ f p
(Well at least that's got rid of the unsafePerformIO :-)
Would this be safe?
It would probably be safer than the version with the unsafePerformIO, but still: foreign import ccall foo :: CString -> Word32 -> Word32 main = do bar <- foo <+ withCString "Hello, world." print (bar 42) ... same problem as above. Cheers, Wolfgang
On Thursday 29 May 2003 16:06, Wolfgang Thaller wrote:
Whoops, now I see a real problem with what I'm asking for.
I guess since the (hypothetical) withCString is creating a new CString (bit like an IORef) it should have type
withCString :: String -> (CString -> b) -> IO b
It's still highly dangerous. When will the string be freed (no you can't rely on the garbage collector, because a CString is a Ptr and not a ForeignPtr; so the string has to be freed explicitly).
If it's implementation was similar to current withCString your right I guess. I was assuming that the (hypothetical) new implementation of withCString was not constrained in this way (I.E. did proper garbage collection on whatever structures it allocated). I don't think it's possible for users to implement this themselves using the FFI as it is (well I haven't managed it), but with some behind the scenes black magic I still think it should be possible (in principle). If this is so, AFAICS the only reason the result type needs to be IO b is because whatever the pointer points to is potentially mutable (like an IORef). So I don't think we can have.. withCString :: String -> (CString -> b) -> b for the same reason that we can't have.. newIORef :: a -> IORef a Maybe we could also have something like this to avoid marshalling the same string again and again.. unsafeWithConstCString :: String -> (CString -> b) -> b unsafeWithConstCString s f = unsafePerformIO $ withCString s f ..but only use this with functions which promised not to mutate the CString.
and I guess my <+ should be.. (<+) :: (a -> b) -> ((a -> IO b) -> IO b) -> IO b f <+ alloc = alloc $ \p -> return $ f p
(Well at least that's got rid of the unsafePerformIO :-)
Would this be safe?
It would probably be safer than the version with the unsafePerformIO, but still:
foreign import ccall foo :: CString -> Word32 -> Word32
main = do bar <- foo <+ withCString "Hello, world." print (bar 42)
... same problem as above.
Oh dear, you're right of course. Still not safe. Sigh. Regards -- Adrian Hey
participants (3)
-
Adrian Hey -
Malcolm Wallace -
Wolfgang Thaller