- FFI - Haskell.org

marshalling parameters (poor/lazy man's IDL)
by Ross Paterson 13 May '04

13 May '04

Suppose you want to call a C function with signature int f(long, double *, float *, short *); where the first parameter is passed by value, the second by reference, the third is an output and the the last is updated. In IDL terms: int f([in]long, [in,ref]double *, [out]float *, [in,out]short *); Now you could just use HDirect, which will write everything for you, but let's suppose you choose not to for some reason. Then you declare the foreign function foreign import ccall "f" c_f :: CLong -> Ptr CDouble -> Ptr CFloat -> Ptr CShort -> IO CInt and you write a Haskell wrapper that does the parameter marshalling by hand: f :: CLong -> CDouble -> CShort -> IO (CFloat, CShort, CInt) f a b d = with b $ \ pb -> alloca $ \ pc -> with d $ \ pd -> do r <- c_f a pb pc pd c <- peek pc d' <- peek pd return (c, d', r) The last bit could be any computation using c, d' and r, but the rest is boilerplate. The plan is to rewrite that as f :: CLong -> CDouble -> CShort -> IO (CFloat, CShort, CInt) f a b d = call c_f (val a . ref b . out . inout d) $ \ c d' r -> return (c, d', r) for suitable combinators call, val (because "in" is a reserved word), ref, out and inout. (You may be reminded of Danvy's printf.) The call combinator takes 3 arguments: - a worker function, usually a foreign function - a specification, consisting of a composition of parameter descriptions in the same order as the function signature, and - a continuation that takes the values of the outputs (again in the same order as they occur in the signature) and the result. The val, ref and inout combinators take the values to be passed; the out and inout combinators give rise to parameters of the final continuation. When value parameters occur first (as they often do), we can move them into the worker function: f :: CLong -> CDouble -> CShort -> IO (CFloat, CShort, CInt) f a b d = call (c_f a) (ref b . out . inout d) $ \ c d' r -> return (c, d', r) There's also a variant call_ that ignores the result: f_ :: CLong -> CDouble -> CShort -> IO (CFloat, CShort) f_ a b d = call_ c_f (val a . ref b . out . inout d) $ \ c d' -> return (c, d') It remains to define the combinators. The following is Haskell 98. > module Param where > import Foreign > import Foreign.C The parameter specifiers build a function that combines a worker function and a continuation operating on the results of the worker, to produce an IO action. Typically each specifier adds a parameter to the worker, and may add one to the continuation. So in general their types look like (rf -> rk -> IO b) -> (ExtraParam -> rf) -> (ExtraResult -> rk) -> IO b Value and reference parameters also take the input value. There is no extra result, and the combinators are fairly simple: > val :: a -> (rf -> rk -> IO b) -> (a -> rf) -> rk -> IO b > val a spec f k = spec (f a) k > ref :: Storable a => a -> (rf -> rk -> IO b) -> (Ptr a -> rf) -> rk -> IO b > ref a spec f k = with a $ \ p -> spec (f p) k In the case of output parameters, the worker takes an extra pointer, and we need to peek at that pointer to get the extra output, while preserving all the arguments that follow it. For that, we need a class that provides a generalized =<<: > class LiftIO r where > liftIO :: (a -> r) -> IO a -> r > instance LiftIO (IO c) where > liftIO = (=<<) > instance LiftIO r => LiftIO (b -> r) where > liftIO f v b = liftIO (flip f b) v and we use this to peek at the pointer: > out :: (Storable a, LiftIO rk) => > (rf -> rk -> IO b) -> (Ptr a -> rf) -> (a -> rk) -> IO b > out spec f k = alloca $ \ p -> spec (f p) (liftIO k (peek p)) Inout parameters combine reference and output parameters: > inout :: (Storable a, LiftIO rk) => > a -> (rf -> rk -> IO b) -> (Ptr a -> rf) -> (a -> rk) -> IO b > inout a spec f k = with a $ \ p -> spec (f p) (liftIO k (peek p)) Finally, 'call' applies this composition of parameter specifications to the base case, which is simply monadic binding of the result continuation to a worker action: > call :: rf -> ((IO a -> (a -> IO b) -> IO b) -> rf -> rk -> IO b) -> rk -> IO b > call f spec_fn k = spec_fn (>>=) f k Here's the variant that ignores the result: > call_ :: rf -> ((IO a -> IO b -> IO b) -> rf -> rk -> IO b) -> rk -> IO b > call_ f spec_fn k = spec_fn (>>) f k There are also other parameter marshalling combinators not used in the above example. A 'Maybe' value passed as either a null pointer or a reference: > unique :: Storable a => > Maybe a -> (rf -> rk -> IO b) -> (Ptr a -> rf) -> rk -> IO b > unique mb_a spec f k = withMaybe mb_a $ \ p -> spec (f p) k > where > withMaybe :: Storable a => Maybe a -> (Ptr a -> IO b) -> IO b > withMaybe Nothing k = k nullPtr > withMaybe (Just a) k = with a k Null-terminated strings: > string :: String -> (rf -> rk -> IO b) -> (CString -> rf) -> rk -> IO b > string s spec f k = withCString s $ \ p -> spec (f p) k Strings followed by their length: > stringLen :: Integral n => > String -> (rf -> rk -> IO b) -> (Ptr CChar -> n -> rf) -> rk -> IO b > stringLen s spec f k = > withCStringLen s $ \ (p, n) -> spec (f p (fromIntegral n)) k Arrays followed by their length: > arrayLen :: (Storable a, Integral n) => > [a] -> (rf -> rk -> IO b) -> (Ptr a -> n -> rf) -> rk -> IO b > arrayLen arr spec f k = > withArrayLen arr $ \ n p -> spec (f p (fromIntegral n)) k

1 0

REMINDER - Contributions to HC&A Report (May 2004 edition)
by Arthur van Leeuwen 03 May '04

03 May '04

Dear Haskellers, we would like to remind you of the opportunity to contribute to the May 2004 edition of the Haskell Communities & Activities Report. Why not take the few minutes necessary to type up a new or updated entry and send it in? We would love to see people write on the following subject(s): - The Haskell Mailing List - The Haskell Cafe - The Haskell 98 reports - Concurrent Haskell - hierarchical libraries - HaskellDB - HXml - QuickCheck - Yale Haskell group - Parallel and Distributed Functional Languages Research Group at Heriot-Watt University Ofcourse, if you have done other interesting things with Haskell we would very much like to know about those as well! If at all possible, could you please send in your contribution by friday *this* week, i.e. May 7th 2004? We will update the topics list on http://www.haskell.org/communities/topics.html with a nice tickmark as soon as we have received your contribution, to show that you too support the Haskell Communities and Activities Report! Thanks, Arthur (interim editor, May 2004 edition) -- Haskell Communities and Activities Report (May 2004 edition) All contributions are due in by the end of next week! http://www.haskell.org/communities/

1 0

Extension to the FFI to allow macro expansions to be avoided
by Ian Lynagh 17 Apr '04

17 Apr '04

[This has come from the cvs-ghc list. All the info should be in this mail, though]. The problem =========== Some C libraries (curses is the one I have to hand) provide macros in addition to functions in some cases for efficiency reasons. If you use the FFI to import such a function thus: ----- module Q where import Foreign.Ptr (Ptr) import Foreign.C.Types (CInt) foreign import ccall unsafe "static curses.h wstandend" wstandend :: Ptr a -> IO CInt ----- Then GHC fails if not using the NCG (which AFAIK is available on only handful of platforms; the generated code efficiency may also be an issue if you don't have enough foreign imports to warrant splitting them off into their own module): $ ghc -fffi -c Q.hs -O /tmp/ghc1164.hc: In function `Q_zdwccall_entry': /tmp/ghc1164.hc:29: warning: dereferencing `void *' pointer /tmp/ghc1164.hc:29: error: request for member `_attrs' in something not a structure or union $ as it is creating code like this (line 29 is the one containing "wstandend"): { I_ _ccall_result; StgAddr _ccall_arg1=_B0_; CALLER_SAVE_SYSTEM _ccall_result = (wstandend((_ccall_arg1))); CALLER_RESTORE_SYSTEM _B3_=_ccall_result; } which then gets expanded to this by cpp: { I_ _ccall_result; StgAddr _ccall_arg1=_B0_; _ccall_result = (((((_ccall_arg1))->_attrs = (0L)))); _B3_=_ccall_result; } (there was some disagreement over whether such a macro was poorly written for not casting its argument; either way, the fact remains that this sort of thing is out there, and convincing the rest of the world to change to make things a bit easier for us is not going to happen.). Possible solutions ================== 1 = One solution is to write a C wrapper around each potentially affected function. I don't think this is very satisfactory as it significantly increases the amount of effort needed to create a binding (rather than just a single foreign import statement you now also need a line in a C file and to make sure the object file is built and linked in correctly). 2 = Surrounding the function with parentheses so the macro doesn't match. The offending line above would be _ccall_result = ((wstandend)((_ccall_arg1))); 3 = I haven't tested this, but I assume you could just generate #undef wstandend before the function call. When to fix the problem ======================= (I'm assuming solution 1 is rejected from here on - if it's the one chosen then the rest is irrelevant) The above gives ways to fix the problem, but there were worries that it could cause a significant performance penalty to always do this, including in some of the bits of GHC implementation. Two possible alternatives come to mind. A = Where a cid is permitted also permit '(' cid ')' B = Before the optional '&' in a C impent, allow an optional "nomacro". The implementation would then make sure you didn't get a macro using solution 2, 3 or some other means. ===== Both A and B allow the programmer to highlight where they cannot trust macros to behave as needed (by ghc -fvia-C at least), so the performance hit is taken only when we can't be sure it is safe not to. ===== In case it isn't clear, I don't like solution 1 (which is what we have at the minute), but would be happy with any combination of {1,2} and {A,B}. Thanks Ian

9 15

RE: Extension to the FFI to allow macro expansions to be avoided
by Simon Marlow 17 Apr '04

17 Apr '04

On 16 April 2004 11:33, Marcin 'Qrczak' Kowalczyk wrote: > Perhaps we should differentiate by a calling convention between > linking to a function directly and generating a source-level C call? > > The calling conventions we know would correspond to linking to them > directly. They would perform the necessary magic to turn off macros > when compiling via C, and they would not need a C compiler when > compiling via assembler. There is no need for specifying a C header > in this case. > > Another calling convention would generate a function wrapper with > proper argument types. It would need "foreign type" declarations and > included headers. The wrapped calling convention in this case is of > course irrelevant, so it's a calling convention rather than a > separate flag. I wondered about that (while writing my last post) but it still seems like a hefty price to pay for the small number of problematic cases we have, and given that you can already write the C stubs by hand to get the desired effect. Hmm, perhaps this falls in to the category of something we expect an external tool to do? Cheers, Simon

2 1

RE: Extension to the FFI to allow macro expansions to be avoided
by Simon Marlow 15 Apr '04

15 Apr '04

> Fergus Henderson wrote: > > [...] the Haskell FFI has sacrificed quality for easy of > implementation. > > If this starts causing problems for users, I think the > right solution > > is to raise the bar for implementations, not to lower it. [...] > > I definitely support Fergus' point of view, GHC and Hugs really rushed > head over heels towards a wrong direction, IMHO. If I see > this correctly, > the only problems were that GHC's native code generator doesn't handle > macros currently and the handling of "broken" macros in the C > back-end and > Hugs. I think we can safely forget about the latter (by using > #undef or > a small wrapper), and we should have a look how to solve the former. > Mercury seems to handle this, although I'm currently not sure at what > price (in terms of implementation effort and possible > performance loss). Fergus, as usual, has a number of good points. I agree that a good FFI should target the C API rather than the ABI, tagetting the ABI is a compromise. The errno example isn't valid here: errno isn't a function, and it isn't guaranteed to be an addressable location either, so the FFI can't access it. There are some other good examples though: on Windows, lots of string functions have two variants, one with an 'A' suffix (ASCII) and one with a 'W' suffix (WIDE), and the plain function name without the suffix is #defined to one or the other based on the value of the UNICODE CPP symbol. The documentation only refers to the un-suffixed version, so in order to write Haskell FFI code that calls these functions you have to dig through the header files, or compile via C, or use a separate C stub. This is really annoying. It's one of those situations where doing The Right Thing hurts you in the pockets, and only fixes a small minority of problematic cases that could be worked around anyway. Worse is better, as they say :-P Cheers, Simon

1 0

RE: Extension to the FFI to allow macro expansions to be avoided
by Simon Marlow 15 Apr '04

15 Apr '04

> On Thu, Apr 08, 2004 at 10:18:24AM +0100, Alastair Reid wrote: > > Now that I understand the problem, my feeling is that the > problem is not with > > curses but with GHC's compilation method. GHC is using a > shortcut by > > pretending that the ffi is for interfacing to cpp+C whereas > the ffi is quite > > clear that it is for interfacing to C. So, I think the > thing to do is fix > > GHC. > > And Hugs too. The issue isn't extending the FFI but implementing it > more accurately and consistently. As you point out, systems compiling > via C have been extending the FFI to a function+macro > interface, which is > incompatible with systems compiling to native code. Having > been bitten by > the same thing in the opposite direction (macros that work > with ffihugs > or ghc -fvia-C don't work with ghc -fasm), I'd favour turning off the > macro interface, preferably with #undef, at least by default. Agreed. Why is #undef to be preferred over adding parentheses around the function name as Ian originally suggested? Contrary to what I first thought, disabling the use of macros in FFI calls should have no impact on GHC, so I'm happy to make this change. Cheers, Simon

3 2

RE: Extension to the FFI to allow macro expansions to be avoided
by Simon Marlow 09 Apr '04

09 Apr '04

> Alastair Reid wrote: > > [...] where the decision to insert parens is decided by a > command-line flag. [...] > > I'm still not convinced that any change is really necessary, > but I'd really > prefer a command-line flag over a change in the FFI syntax. > The latter will > lead to much more portability trouble than the former. Adding a flag is fine by me. Cheers, Simon

1 0

RE: Request: withArrayLength
by Simon Marlow 30 Mar '04

30 Mar '04

> The suggestion wasn't to rename it, but to move it from base > to haskell98. > > Then you could add these functions without changing the > interface of any > of the modules listed in the FFI Addendum. That would amount to saying that 'Foreign' isn't a hierarchical library any more, and anyone wishing to use purely hirerarchical libraries in their code should avoid it. On one hand, this looks reasonable: Foreign already doesn't re-export Foreign.C. However, I do use it all the time. So if we ever decided to allow disabling of the haskell98 package, lots of supposedly purely-hierarchical code would break. Ho hum. Cheers, Simon

1 0

RE: Request: withArrayLength
by Simon Marlow 30 Mar '04

30 Mar '04

> I think we need to decide how the further development of the FFI will > proceed. Given that Version 1.0 of the Addendum is frozen, I > am not in > favour of changing the Addendum anytime soon and certainly > not for small > matters, such as a convenience function. > > We might want to accumulate extra useful functions in the > implementation > in an *extra* module and once there is enough interesting > functionality > in there, we can start talking about Version 1.1 of the Addendum. > > Does this sound like a sensible approach? We certainly want to be able to add new stuff to the FFI over time. There are several ways we might do this: (1) Just extend the existing interfaces (don't change semantics of existing functions, though). (2) Add new modules only (not re-exported by Foreign) (3) Add new stuff to the hierarchical Foreign.XXX modules, but not the non-hierarchical variants (and Foreign keeps the same interface as the FFI spec). (4) Wait until we have versioned packages, and have separate ffi-1.0 and ffi-1.1 packages. I like (4) the best, but we can't do that yet - we know how to, but there's a fair bit of work to get there. Of the other options, I'm tempted by (1). (2) is unattractive because some of the new interfaces will morally belong in existing modules (eg. withArrayLength clearly belongs in Foreign.Marshal.Array). (3) is unattractive because the Foreign module overlaps, so we can't keep the hierarchical and non-hierarchical versions separate. (1) is unattractive because it might break code that works with the FFI 1.0 spec (unlikely, though). Cheers, Simon

5 7

RE: Request: withArrayLength
by Simon Marlow 24 Mar '04

24 Mar '04

> I think adding new (hierarchical) modules is the way to go for now. > > 1) Hierarchical because it lets us use some nice clear > module naming style like: > > Foreign.NonStd.* > or > Foreign.Extension.* > > [I favour the former since the status of the lib is clearer.] Not wanting to throw a spanner in the works, but we did agree not to use these kinds of names back in the early days of the hierarhcical libraries. The general rule is to base names on the functionality provided by the libraries only. There are a few cases where we break the rules at the moment: System.Posix and System.Win32 are a bit dubious, and there's Unstable.Control.Monad. But on the whole we're sticking to the policy, and there are good reasons for doing so, so I vote not to introduce Foreign.NonStd. I'm beginning to think we should implement package versioning ASAP. We would need support for it in all compilers, though. Cheers, Simon

1 0