- FFI - Haskell.org

qforeign-0.63 is out
by qrczak＠knm.org.pl 29 Dec '00

29 Dec '00

http://download.sourceforge.net/qforeign/qforeign-0.63.tar.gz (Will be in CVS there soon.) Changes since 0.62: * Adapted to MarshalUtils and ForeignPtr introduced in ghc-4.11. * Works on ghc-4.08 through ghc-4.11 and nhc98-1.00 through nhc98-1.01. Uses native modules where available, otherwise supplies missing parts itself if possible. (nhc is currently too buggy and produces crashing programs.) * Examples split into separate packages which link the right C libraries and #include the right C headers. Ready for using in other people's programs. * qforeign-config script for using with ghc versions without user packages and with nhc. * hsc2hs (renamed from glue-hsc) implements #undef, #error, and #let (macros to be applied to the Haskell source, although using somewhat ugly stringified syntax). * Can be used either after 'make install' or after building inplace. * Db3 interface started (very incomplete). * Include files CHANGES and COPYING. -- __("< Marcin Kowalczyk * qrczak(a)knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK

1 0

Re: cvs commit: fptools/hslibs/lang ForeignPtr.lhs MarshalUtils.lhs Foreign.lhs ForeignObj.lhs Storable.lhs
by qrczak＠knm.org.pl 12 Dec '00

12 Dec '00

Mon, 11 Dec 2000 02:03:27 -0800, Simon Marlow <simonmar(a)glass.cse.ogi.edu> pisze: > Change to the FFI recently discussed on the mailing list. > Still to come: error handling utilites. > > This is mostly untested. Thanks. I've been sceptic about ForeignPtr, but now I like it. In three of qforeign's library interfaces it is applied only to type tags, but it caused a small simplification: no newtype constructor wrapping and unwrapping. Ptr () in foreign imports changed into Ptr TheAppropriateTypeTag, which is good. I've been worried that in practice types become non abstract (of course one could always use newtypes, but having both a newtype and a tag is somewhat redundant). But it's not really a problem. One requires ghc extensions to make instances for such types, but the whole FFI is an extension anyway. OTOH when a different (==) than pointer equality is needed, the type must still be wrapped in newtype. In the fourth case, Libgr, it was a real improvement which convinced me that ForeignPtr was a good idea. Parametrized newtyped ForeignObjs used in overloaded functions, which required using pattern and result type signatures to pass the type from the parameter of the newtype to pointers extracted from the ForeignObj, changed into ForeignPtr applied to real storable types. All types magically began fitting. It is similar to the case of character encoding conversions, where I originally wanted them to have a single type but distinguishing streams of Chars from streams of Bytes caused surprising simplifications. I learned that types acting as tags for parametrizing other types are really nice, and Haskell likes to be statically typed! If only it was not necessary to introduce value constructors in definitions of type tags... -- __("< Marcin Kowalczyk * qrczak(a)knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK

1 0

RE: extended foreign decls
by Simon Peyton-Jones 07 Dec '00

07 Dec '00

| For ("preferred" language, foreign language) pair, you define an | "extrusion". This defines how to generate the appropriate code to | interface with one foreign language in another. All sounds very ingenious. Much of it, though, is quite Mercury independent. As you say, it's a way of making the Mercury compiler think it's only talking to C. Obvious question: do you plan to package your solution as a reusable tool, so that others can profit from its ingenuity? Simon

2 1

RE: Proposal #2: marshalling utilites
by Simon Marlow 07 Dec '00

07 Dec '00

> Simon Marlow <simonmar(a)microsoft.com> wrote, > > [...] With that in mind, I'll propose the following changes: > > [...] > > I agree with Manuel here: > > * Basically a "Make it so!" > > * Error handling should go into another module, but I'll > stay off the naming battlefield this time. :-) > > * errno *is* a thread local thing, of course. Not a Haskell thread local thing, unfortunately. I *really* don't want to have a private errno in each Haskell thread object, which gets copied in after every foreign call. Anyone have any bright ideas here? > * s/Marshal/MarshalUtils/ ok, that seems to be the consensus. Cheers, Simon

4 3

FW: H/Direct
by Simon Peyton-Jones 06 Dec '00

06 Dec '00

Here's a question from Koen that bears on the FFI. Simon -----Original Message----- From: Koen Claessen [mailto:koen@cs.chalmers.se] Sent: 04 December 2000 12:55 To: Erik Meijer; Simon Peyton-Jones; Sigbjorn Finne Subject: H/Direct Hi H/Direct team, I have been using H/Direct for quite a while now, and I am a happy user (apart from the massive amount of work involved in getting the system to run for both Hugs and GHC -- there seems to be a circular import which Hugs complains about and which can be skillfully removed, but which GHC depends on or something.) I have a question about marshalling arguments which are expensive to marshall in H/Direct. Often you want this result to be shared between calls of the functions. Currently, I have applied the following technique a couple of times and I wonder if it can be made more general and added to H/Direct. For example, I have a BDD package with the following operation (I have simplified this type a bit): [pure]Bdd *BddReplace ( [size_is(n),in] int *xs , int n , [in] Bdd *bdd ); (The argument "xs" encodes a variable mapping, which is used to construct a new BDD from the argument.) I want this to be a function in Haskell with the type: replace :: [Int] -> Bdd -> Bdd The way I often use replace is by defining: prime :: Bdd -> Bdd prime = replace [ extremely-long-list ] Here, I want the marshalling of the list to happen once, and to be shared between calls to prime. So I wrote my own version of replace in Haskell. Here it is: replace :: [Int] -> (Bdd -> Bdd) replace xs = unsafePerformIO $ do array <- newCArray (length n) sequence [ setCArray array i x | (i,x) <- [0..] `zip` xs ] return (\bdd -> primBddReplace bdd array) Note the brackets in the type to indicate I am returning a function. Now, the CArray interface must look like this: [finaliser("free")] interface CArray {}; CArray *newCArray( int n ); void setCArray( CArray *arr, int i, int n ); int readCArray( CArray *arr, int i ); This is because "replace" should not free the C-array until it looses track of the returned function. It works great! The question is: can this be done by inventing some new annotation for the IDL code? One problem is maybe that the sharing of these things can only be done in a certain order. Regards, Koen. -- Koen Claessen http://www.cs.chalmers.se/~koen phone:+46-31-772 5424 mailto:koen@cs.chalmers.se ----------------------------------------------------- Chalmers University of Technology, Gothenburg, Sweden

1 0

RE: efficiency
by Simon Marlow 05 Dec '00

05 Dec '00

> > because I recon eliminating the exception handler is more > important than > > speeding up the actual malloc/free. > > Hmm... alright, here's some quick benchmarks using 4.08.1: 1e6 * "bracket" = 0.20 secs 1e6 * "bracket+malloc/free" = 0.69 secs 1e6 * "malloc/free" = 0.40 secs 1e6 * "allocate ByteArray" = 0.13 secs Ok, so the malloc/free is the most expensive part. I'm moderately surprised :) Perhaps we could do a better job, but we'll never do better than ByteArrays as long as the bracket is required. Cheers, Simon

5 6

Proposal #2: marshalling utilites
by Simon Marlow 05 Dec '00

05 Dec '00

The C strings stuff is still up in the air, so I'll leave that for the time being. The main concern as far as adding functionality to the marshalling libraries seems to be the need to separate the compiler-dependent primitives from the rest of the library. With that in mind, I'll propose the following changes: * Deprecate everything in Storable except the class itself and its instances. * Add the Marshal module below (the names are of course all up for discussion, including the name of the module itself). The implementation of Marshal is compiler independent, except there's a requirement for a way to convert an errno value into an IOError. As noted below, perhaps the error processing, or at least the compiler-dependent portion, should be farmed off into another module. I omitted some of the generic utilities from Marcin's proposal, since while obviously useful they don't seem to belong here. Note that this isn't supposed to be the "higher-level marshalling library" we've been talking about, it's just s small step from what we have now, but an important step in that it puts the FFI into a usable state (at least for interfacing to C libraries, modulo the C strings support). That's important for us in the GHC team at least because it means we can start using this in GHC's libraries. My take on the C strings stuff: we can, and should, provide a portable-ish implementation of poke/peekCString which uses a (fictitious?) conversion library to do the character conversions, and uses standard marshalling to get to/from [CChar]. The point is that we'll probably need to optimise this at least in GHC, both to avoid writing our own conversion code and to speed things up, so it makes sense to place it all behind a specialised set of C string marshalling calls. Cheers, Simon module Marshal ( ------------------------------------------------------------------------ -- Allocation, deallocation and marshalling for objects and arrays malloc, -- :: Storable a => IO (Ptr a) mallocArray, -- :: Storable a => Int -> IO (Ptr a) mallocBytes, -- :: Int -> IO (Ptr a) alloca, -- :: Storable a => (Ptr a -> IO b) -> IO b allocaArray, -- :: Storable a => Int -> (Ptr a -> IO b) -> IO b allocaBytes, -- :: Int -> (Ptr a -> IO b) -> IO b reallocArray, -- :: Storable a => Ptr a -> Int -> IO (Ptr a) reallocBytes, -- :: Ptr a -> Int -> IO (Ptr a) free -- :: Ptr a -> IO () pokeArray, -- :: Storable a => Ptr a -> [a] -> IO () pokeArray0, -- :: Storable a => a -> Ptr a -> [a] -> IO () peekArray, -- :: Storable a => Ptr a -> Int -> IO [a] peekArray0, -- :: (Storable a, Eq a) => a -> Ptr a -> IO [a] withObject, -- :: Storable a => a -> (Ptr a -> IO b) -> IO b withArray, -- :: Storable a => [a] -> (Ptr a -> IO b) -> IO b withArray0, -- :: Storable a => a -> [a] -> (Ptr a -> IO b) -> IO b new, -- :: Storable a => a -> IO (Ptr a) newArray, -- :: Storable a => [a] -> IO (Ptr a) newArray0, -- :: Storable a => a -> [a] -> IO (Ptr a) ---------------------------------------------------------------- -- Haskellish interface to memcpy and memmove. copyArray, -- :: Storable a => Ptr a -> Ptr a -> Int -> IO () copyBytes, -- :: Ptr a -> Ptr a -> Int -> IO () moveArray, -- :: Storable a => Ptr a -> Ptr a -> Int -> IO () moveBytes, -- :: Ptr a -> Ptr a -> Int -> IO () ------------------------------------------------------------------ -- Error handling. -- Perhaps these should be moved to another module (PosixError?) getErrno, -- :: IO Int throwIf, -- :: (a -> Bool) -> String -> IO a -> IO a throwIf_EINTR, -- :: (a -> Bool) -> String -> IO a -> IO a throwIfNull, -- :: String -> IO (Ptr a) -> IO (Ptr a) throwIfNull_EINTR, -- :: String -> IO (Ptr a) -> IO (Ptr a) throwIfMinus1, -- :: Num a => String -> IO a -> IO a throwIfMinus1_EINTR, -- :: Num a => String -> IO a -> IO a throwCError, -- :: String -> Int -> IO a throwErrno, -- :: String -> IO a ------------------------------------------------------------------------ ----- -- Misc indexPtr -- :: Storable a => Ptr a -> Int -> Ptr a )

4 3

Proposal #1: ForeignPtr
by Simon Marlow 05 Dec '00

05 Dec '00

Dear FFI folks, I'd like to summarise the current proposals on the table and see if we can get a consensus. Proposal #1: replace ForeignObj with (ForeignPtr a). * new module ForeignPtr, with the following interface: module ForeignPtr ( ForeignPtr, -- abstract, instance of: Eq , newForeignPtr -- :: Ptr a -> IO () -> IO (ForeignPtr a) , addForeignPtrFinalizer -- :: ForeignPtr a -> IO () -> IO () , withForeignPtr -- :: ForeignPtr a -> (Ptr a -> IO b) -> IO b , foreignPtrToPtr -- :: ForeignPtr a -> Ptr a , touchForeignPtr -- :: ForeignPtr a -> IO () , caseForeignPtr -- :: ForeignPtr a -> ForeignPtr b ) * deprecate all of ForeignObj * replace references to ForeignObj in the FFI spec with ForeignPtr. * export module ForeignPtr from Foreign. * GHC only: deprecate Weak.addForeignFinalizer Cheers, Simon

3 2

qforeign-0.62
by qrczak＠knm.org.pl 05 Dec '00

05 Dec '00

An updated version is at <http://qrczak.ids.net.pl/qforeign-0.62.tar.gz>. Major changes from 0.60: * 'make install' implemented. Installs as package qforeign. * iconv support is more robust. Works with Konstantin Chuguev's iconv implementation too. * Definitions of character predicates are more correct, after discussions on linux-utf8 and unicode mailing lists. * glue-hsc: #const_str construct added * Curses interface improved. * Pcap interface added (incomplete). * Works under ghc-4.08, 4.08.1, 4.09 (before-ghci-branch). No Unicode or 'make install' with ghc < 4.09 (I just need to use FFI with ghc-4.08*). * Things that went into ghc are removed from here (Ptr, Storable). glue-hsc from here is used only for ghc < 4.09. ------------------------------------------------------------------------ qforeign got smaller because of changes in ghc, but what is ghc is still a bit incomplete. Most important incompleteness is in string handling. Since ghc's Unicode support is half-baked and it is not yet clear how the library interface will look like, I propose adding the following functions to the module Foreign: peekCString :: Ptr CChar -> IO String peekCStringLen :: Ptr CChar -> Int -> IO String withCString :: String -> (Ptr CChar -> IO a) -> IO a withCStringLen :: String -> (Ptr CChar -> Int -> IO a) -> IO a newCString :: String -> IO (Ptr CChar) newCStringLen :: String -> IO (Ptr CChar, Int) -- Memory is obtained by malloc and must be freed using free. Unfortunately versions with explicit length need to be separate. Most code will use those without the length, but in case a C library allows handling strings with '\0' characters embedded by passing the length explicitly, the more common versions are not enough. Their implementation will use ISO-8859-1 for now. Their interface is not constrained to the case where the length of the C string translated from Haskell is the same as of the original, nor vice versa. Eventually they will do the conversion between Unicode and the default local encoding and there will be separate versions allowing to supply another conversion. ------------------------------------------------------------------------ After adding the above, the interface provided by ghc alone should be sufficient to do almost everything currently possible with FFI, without the need of using private ghc's bits and with reasonable efficiency. Apart from that, my library contains some convenience functions. I hope the meaning is clear from names and types. Perhaps some of them are worth inclusion into ghc? ---------------------------------------------------------------- -- Storable arrays. No new expressiveness, but they are commonly -- used. peekArray :: Storable a => Ptr a -> Int -> IO [a] peekArray0 :: (Storable a, Eq a) => a -> Ptr a -> IO [a] pokeArray :: Storable a => Ptr a -> [a] -> IO () pokeArray0 :: Storable a => a -> Ptr a -> [a] -> IO () withObject :: Storable a => a -> (Ptr a -> IO b) -> IO b withArray :: Storable a => [a] -> (Ptr a -> IO b) -> IO b withArray0 :: Storable a => a -> [a] -> (Ptr a -> IO b) -> IO b newObject :: Storable a => a -> IO (Ptr a) newArray :: Storable a => [a] -> IO (Ptr a) newArray0 :: Storable a => a -> [a] -> IO (Ptr a) -- Memory is obtained by malloc and must be freed using free. ---------------------------------------------------------------- -- Haskellish interface to memcpy and memmove. copyArray :: Storable a => Ptr a -> Ptr a -> Int -> IO () copyBytes :: Ptr a -> Ptr a -> Int -> IO () moveArray :: Storable a => Ptr a -> Ptr a -> Int -> IO () moveBytes :: Ptr a -> Ptr a -> Int -> IO () ---------------------------------------------------------------- -- Lifting many functions taking a continuation (which takes a value) -- to a single function taking a continuation (which takes a list). -- This is tricky to write each time by hand. -- Convenient for sequencing alloca and with functions. -- For example to allocate an array of C strings. sequenceCont :: [(a -> res) -> res] -> ([a] -> res) -> res mapCont :: (a -> (b -> res) -> res) -> [a] -> ([b] -> res) -> res ---------------------------------------------------------------- -- Perhaps unsafePerformIO and unsafeInterleaveIO should be -- reexported from module Foreign too. Foreign import may import -- pure functions anyway, but in that case the wrapper cannot do -- memory allocation etc. unsafePerformIO :: IO a -> a unsafeInterleaveIO :: IO a -> IO a ---------------------------------------------------------------- -- Handling errors. Mostly conversion of errno to Haskell's IO -- exceptions. -- Get the current value of C errno. getErrno :: IO Int -- Throw an exception based on function name and given error code. throwCError :: String -> Int -> IO a -- Throw an exception based on function name and current errno -- value. Much more common than the above but less general, -- that's why there are two functions. throwErrno :: String -> IO a -- Perform an action and ignore its result. A stupid thing but quite -- common, especially after the result was converted to exceptions. ignore :: IO a -> IO () -- Wrappers that convert the result to a possible exception. -- *_EINTR variants repeat the action if there was an error -- and errno == EINTR. throwIf :: (a -> Bool) -> String -> IO a -> IO a throwIf_EINTR :: (a -> Bool) -> String -> IO a -> IO a throwIfMinus1 :: Num a => String -> IO a -> IO a throwIfMinus1_EINTR :: Num a => String -> IO a -> IO a throwIfNull :: String -> IO (Ptr a) -> IO (Ptr a) throwIfNull_EINTR :: String -> IO (Ptr a) -> IO (Ptr a) ---------------------------------------------------------------- -- Mutexes. They allow making non-reentrant functions thread-safe, -- without importing the whole Concurrent stuff. They are also -- implementable in a Haskell compiler without concurrency, and -- that's why I think they should be exposed by the FFI instead of -- using Concurrent things directly. -- errno handling is not reentrant, even though it is reentrant -- in native C threads. Instead of protecting all wrappers for C -- functions which use errno by mutexes, I propose that Haskell -- compilers which provide concurrency should be responsible for -- maintaining a thread-local C errno variable. data Mutex newMutex :: IO Mutex threadSafe :: Mutex -> IO a -> IO a -- __("< Marcin Kowalczyk * qrczak(a)knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK

4 8

RE: efficiency
by Simon Marlow 04 Dec '00

04 Dec '00

> This is also my main efficiency concern. Would it be an > option to handle thread stacks differently in the memory > manager? Apart from them moving around, thread stacks are > the natural place to get alloca memory from. Thread stacks fall into the category of "large objects" in GHC's storage manager, and therefore aren't normally moved. However, they have to be able to grow on demand, so we occasionally have to relocate them. I suppose we could pre-allocate large stacks if we were planning on doing any static stack allocation, but that seems to be a particularly overgrown path to be wondering down. > I am not sure why you are saying that a static area of > memory allocated in a stack discipline wouldn't be a huge > win? It would ceratinly get rid of the free list handling > of malloc/free. As for exceptions, I am not familiar enough > with their implementation. But we might have to have such > an area per thread... because I recon eliminating the exception handler is more important than speeding up the actual malloc/free. No figures, just compiler hacker's instinct :) You wouldn't be able to eliminate the exception handler if the allocation were done in a separate static area, even if it was allocated stackwise. We've been talking about speeding up catch (and block/unblock) for a while, but haven't done anything yet. Cheers, Simon

2 1