January 2001 - FFI - Haskell.org

RE: Let's get this finished
by Simon Marlow 15 Jan '01

15 Jan '01

> On Mon, 15 Jan 2001, Simon Marlow wrote: > > > data UnsafeCString > > withUnsafeCString :: String -> (UnsafeCString -> IO a) -> IO a > > > > where an UnsafeCString is valid only in an argument position of an > > unsafe foreign import. > > Much of the conversion stuff would have to be done in a different way. > An additional internal interface to conversions implemented in C > (with ByteArrays instead of Ptrs), pointer arithmetic moved from > Haskell to C, and in future hard to mix with Handle I/O done on > Ptrs. Hmm. You're right, unfortunately :( This is depressing. > > I still can't think of a good way to do this in general. Perhaps > > enhancing the garbage collector so that it could "pin" objects - but > > you've still got the problem of keeping the lifetime of the > ByteArray > > in sync with the Ptr. > > Lifetime is not a problem: > withByteArray :: ByteArray -> (Ptr a -> IO b) -> IO b > > (And now we know that the ByteArray type should be parametrized!) Well, the reason I said this was a problem was because I was trying to avoid the overhead of the exception handler or continuation in withByteArray. Actually we can do this right now: you allocate a large ByteArray (>= 4k ish) to make sure it gets allocated in immovable space, and then use touch# to keep it alive. You get through a lot of memory this way, but it should be faster than malloc/free. I'll do some experiments. Cheers, Simon

1 0

RE: Let's get this finished
by Simon Marlow 15 Jan '01

15 Jan '01

> I am all for moving the old CString routines to > UnsafeCString, but that might annoy George and others who > still have code importing the old CString. > > The old code contains cruft like > > fill arr n [] = > _casm_ ``((PP_)%0)[%1] = NULL;'' arr n > fill arr n (x:xs) = do > barr <- packStringIO x > _casm_ ``((PP_)%0)[%1] = (P_)%2;'' arr n barr > fill arr (n+1) xs > > which doesn't make much sense keeping around with the new > FFI stuff in place. It doesn't even look like it works! If I'm reading the above code correctly, it seems to build a ByteArray in which each element is a pointer to another ByteArray, so if any of the arrays moves you're hosed. The Posix library uses it to implement exec. George, are you using it at all? Most uses of unpackCString and friends can be replaced with peekCString (CStrings from a foreign function are bound to be Ptr CChar). For packString, I've been using the following interface instead (in ghc/lib/std/PrelCString): data UnsafeCString withUnsafeCString :: String -> (UnsafeCString -> IO a) -> IO a where an UnsafeCString is valid only in an argument position of an unsafe foreign import. Would anyone object to adding these to CString? > As 4.08.2 will still have the old interfaces, one possible > route might be a more radical step for the first GHCi > release (which will break all kinds of things anyway, I am > sure :-} We could collect all the old interfaces (I guess, > there are other modules beside CString that can be phased > out) in a new module "LangDeprecated" where we maybe even > redefine the old routines in terms of the new ones. Code > using the old interfaces can, then, just import > LangDeprecated until the code gets revised, but it is clear > that the old interfaces will vanish one day. Sounds reasonable, but a lot of hassle to get right. I'm not in favour of hassle purely for the purposes of deprecated interfaces :) > Moreover, as soon as the unicode stuff gets in, all the old > C-style string handling will be wrong anyway. > > As for bytearrays: AFAIK the only reason for using byte > arrays over malloc is efficiency. So, that makes them, I > think, mainly interesting for allocating temporary storage, > which is maybe even more critical for routines that need > pointers to storage as out parameters to C routines doing > little work. This is actually more critical for routines > handling small data items like ints than for strings. So, > we would like to have something like > > with[Object]Unsafe :: Storable a => a -> (Ptr a -> IO b) -> IO b > withArrayUnsafe :: Storable a => [a] -> (Ptr a -> > IO b) -> IO b > withArray0Unsafe :: Storable a => a -> [a] -> (Ptr a -> > IO b) -> IO b > withCStringUnsafe :: String -> (CString -> IO a) -> IO a > withCStringLenUnsafe :: String -> (CStringLen -> IO a) -> IO a > > However, the real problem here is that if we want to use > `ByteArray' instead of `Ptr', the types have to change and > it is not H98 anymore. So, the problem is getting the > pointer from the allocation to the C function without moving > it. *sigh* I still can't think of a good way to do this in general. Perhaps enhancing the garbage collector so that it could "pin" objects - but you've still got the problem of keeping the lifetime of the ByteArray in sync with the Ptr. Cheers, Simon

3 2

qforeign-0.64 is out
by Marcin 'Qrczak' Kowalczyk 15 Jan '01

15 Jan '01

http://download.sourceforge.net/qforeign/qforeign-0.64.tar.gz * Adapted to FFI modules introduced in recent ghc-4-11. * For nhc98 its recent version from CVS is required, but it finally works. * hsc2hs processes # characters only outside comments and string literals, processing a file twice works (.hscc -> .hsc -> .hs), removed #option construct, added #enum construct, added -I, --include, and --version parameters, appropriate LINE pragmas are generated. * hsc2hs documentation is back. * Use one Makefile for all subdirs. (I've read <http://www.pcug.org.au/~millerp/rmch/recu-make-cons-harm.html>). * Check compiler abilities by compiling or running test programs instead of relying on its version. hsc2hs in ghc-4.11 is up to date. -- Marcin 'Qrczak' Kowalczyk

1 0

RE: Let's get this finished
by Simon Marlow 11 Jan '01

11 Jan '01

> Granted. And remember we already have a (quite ugly) module CString! > So here is a new proposal: > > * Deprecate the old definitions in CString and add > > type CString = Ptr CChar > mallocCString :: String -> IO CString > peekCString :: CString -> IO String > pokeCString :: CString -> String -> IO () > withCString :: String -> (CString -> IO a) -> IO a > newCString :: String -> IO CString > > * Add a new module CStringLen with > > type CStringLen = (CString, Int) > mallocCStringLen :: String -> IO CStringLen > peekCStringLen :: CStringLen -> IO String > pokeCStringLen :: CStringLen -> String -> IO () > withCStringLen :: String -> (CStringLen -> IO a) -> IO a > newCStringLen :: String -> IO CStringLen Ok. I'm still using the existing CString, but it can be renamed to something else. I've been using "UnsafeCString" since these ByteArray CStrings can only be passed to unsafe foreign functions. For now, let's just rename CString to UnsafeCString and change all the uses. Cheers, Simon

2 1

RE: PackedString
by Simon Marlow 10 Jan '01

10 Jan '01

> > > Is anybody actually depending on the old `CString'? Or is > > > everybody using `PackedString' instead? > > > > I don't believe anyone is using PackedString, except for Regex which > > arguably should be using CString instead. I'd like to see a simpler > > PackedString which just corresponded to ByteArrays. > > The nhc98 compiler uses PackedString extensively. The interface is > very standard - all of ghc, hbc, and nhc98 support it. Change the > implementation by all means if you wish, but not the interface. Not a problem. I wasn't suggesting changing the interface, sorry for the confusion. Cheers, Simon

1 0

RE: Let's get this finished
by Simon Marlow 10 Jan '01

10 Jan '01

> Is anybody actually depending on the old `CString'? Or is > everybody using `PackedString' instead? I don't believe anyone is using PackedString, except for Regex which arguably should be using CString instead. I'd like to see a simpler PackedString which just corresponded to ByteArrays. Cheers, Simon

3 2

RE: Let's get this finished
by Simon Marlow 10 Jan '01

10 Jan '01

> "Manuel M. T. Chakravarty" wrote: > > If hslibs is meant to be used with systems other than ghc, > > too - which I think was the idea - there is no choice but to > > rewrite it into H98. But I guess this is essentially up to > > Mr. HsLibs aka SimonM. > > Well, doing some simple local transformations to get H98 compliance > should be possible without consulting Mr. HsLibs in advance. :-) [ blimey. I go away for a couple of days and you guys are going bananas. Slow down! ] I should say that I've recently been working on moving most of the FFI code into ghc/lib/std, removing GHC's use of deprecated features (eg. using Ptr instead of Addr), and converting some of GHC's standard libraries to use the new FFI. Directory has been converted so far. I think there's no alternative to having two copies of much of the code - a H98 version in hslibs/lang and GHC-specific code partly in ghc/lib/std and partly in hslibs/lang, with appropriate #ifdefs. We'll need to set up a way to test the H98 code using GHC too. I don't see any problem with using hsc2hs for the H98 versions, since hsc2hs is mostly compiler-independent. (but I notice you've used autoconf instead of hsc2hs for the errno code. I haven't looked at it yet.) Cheers, Simon

2 1

RE: Let's get this finished
by Simon Marlow 10 Jan '01

10 Jan '01

> > [ MarshalError ] > > But SimonM said that he doesn't want to make errno Haskell > > thread local... > > Wasn't this only due to efficiency reasons? IIRC Fergus had a good > suggestion (copying errno only on thread switches or something), so > I'd like to hear Mr. HsLibs' opinion on this topic again. It'll have to be done (copying errno on thread switches, that is). But I still hate it. > > [ MarshalUtils ] > > BTW, can't we just use `with' now. IIRC, the only reason > > Marcin used `withObject' was because of a clash with the > > implicit parameter extension. This has been changed now, > > hasn't it? > > Nope, not yet. Although there was some consensus about SPJ's > ?-suggestion for those bindings, it's not implemented yet. > Any takers? If this has been done, `with' would be better, indeed. Agreed. > > > And just a final topic which really should be settled: > > > "Marshal" or "Marshall"?? :-} > > > > I would have taken the latter, but I think, SimonM > > originally proposed the name MarshalUtils with one "l" and > > as he is the native speaker... dictionary.com recons that 'Marshal' is the correct spelling. 'Marshall' is a military rank only, AFAICT. Cheers, Simon

1 0

Let's get this finished
by Manuel M. T. Chakravarty 10 Jan '01

10 Jan '01

The major piece missing from the FFI library is the high-level marshalling support. Moreover, `MarshalUtils' isn't quite there. In fact, I think, we should have a family of MarshalXXX modules, which include what currently is in MarshalUtils plus some more stuff. MarshalUtils ~~~~~~~~~~~~ The reason that I don't like MarshalUtils as it is are the following: (1) It isn't H98 and in fact its interface cannot be implemented with H98[1]. The problem are the routines with signatures in patterns: malloc :: Storable a => IO (Ptr a) malloc :: IO (Ptr a) = mallocBytes (sizeOf (undefined::a)) alloca :: Storable a => (Ptr a -> IO b) -> IO b alloca (act :: Ptr a -> IO b) = allocaBytes (sizeOf (undefined::a)) act <etc> (2) pokeArray, peekArray, etc are really part of what should go into the high-level marshalling. Point (2) is easily resolved by defining a family of modules including all the necessary functionality. However, Point (1) is a problem. First of all, I guess, we have to revert to the signature for allocation that I originally used in the C2HS library: malloc :: Storable a => a -> IO Addr malloc dummy = mallocBytes (sizeOf dummy) All the other allocation routines have to be altered in a similar way. A similar problem arises for copyArray and moveArray, where I guess the additional argument is in practice more awkward as for the allocation routines. Finally, indexPtr is affected, which is especially annoying since it can't be used infix anymore after the change. The Marshal modules ~~~~~~~~~~~~~~~~~~~ I propose the following set of modules: - MarshalAlloc: The allocation routines that are in MarshalUtils now, but modified as outlined above. - MarshalArray: The array marshalling routines - MarshalString: Most people seem to be of the opinion that we have to handle strings specially due to Unicode conversion requirements. So, this is were string marshalling goes. - MarshalError: Combinators to test for common error conditions. The main open problem here is the handling of errno, I think. - MarshalConv: The C2HS IntConv, BoolConv, etc classes. Marcin just uses `fromIntegral' here, but I think that this is too limited - eg, it doesn't handle Bool. Moreover, soemthing along the C2HS ToAddr/FromAddr classes, but it probably makes to have `stdArrayPtr' etc as additional member functions (this is similar to the newXXX routines that are in MarshalUtils now, but with additional overloading). Furthermore, Marcin-syle withXXX routines would fo here. - MarshalUtils: The memcpy/memmove stuff, indexPtr (if it still makes sense), and whatever else doesn't get its own module. I think, we should leave C2HS's marshX routines out. We discussed how to make them arity independent for quite a while and AFAIR, there wasn't a really good proposal. Finally, all the above modules should be jointly exported by a module `Marshal'. Comments? Cheers, Manuel [1] Which unfortunately escaped me when we discussed MarshalUtils for the first time.

5 27

Re: nhc98-1.{00,01} produce crashing programs
by Malcolm Wallace 08 Jan '01

08 Jan '01

[ Re: FFI in nhc98 - followups set to ffi(a)haskell.org ] > > Int/Word 8/16/64 are not really supported properly at all right now. > > With QForeign they work well enough that the Curses interface works. > QForeign provides Num/Integral/etc. instances, and Int64/Word64 are > not used much. Yesterday, I commited lots of code for Int/Word 8/16/32/64. Some work is still required for 64-bit value conversions only. > I miss foreign export dynamic the most. This is the trickiest part to implement, of course. As far as I know, we can lift the most difficult sections directly out of Hugs or ghc's implementation and re-use it. (But see further comment about foreign import dynamic below.) > Then the fact that case does > not work for arbitrary numeric types. I don't understand - can you explain? Surely the Num instances for Word/Int 8/16/32/64 mean that a literal integer 3 in the source code becomes (fromInteger 3) in reality, and this should be fine in case comparisons. > Then foreign label and foreign import dynamic. Foreign label is probably not too difficult to get going quickly. As to dynamic import and export, I have never had any luck playing with dynamic loading of libraries in C. Can anyone recommend a good tutorial? The manual pages I have read so far were not very enlightening. I'm afraid that, until I understand how to get dynamic libraries going in C, there will be no provision for them in nhc98. > addForeignPtrFinalizer is not emulated under nhc because it does not > provide addForeignFinalizer :: ForeignObj -> IO () -> IO () > (this function is not so important). Specifying a finalizer as IO () (i.e. as Haskell code rather than C code) is not a straightforward idea at all. Currently in nhc98, such finalizers are recorded, but may never be run. This will be fixed eventually. However, as far as I am aware, we agreed to remove the idea of *multiple* finalizers from the common FFI spec, because they have too many potential semantic problems. > QForeign comes with part-of-Parsec and GetOpt from ghc, which are > needed by it but not distributed with nhc. The next release of nhc98 will include GetOpt, Parsec, and a whole host of other libraries. > It also emulates ghc's Bits module with Bits class; nhc's Bit is different. Yes, nhc98's Bit library follows version 1.3 of the Haskell Library Report, whereas ghc's Bits library has never been standard. :-) Fortunately, the next release of nhc98 can provide both interfaces if you wish. > nhc does not provide CTypes and CTypesISO modules, nor HsFFI.h CTypes and HsFFI.h exist as of yesterday. CTypesISO will follow shortly. > 8-bit Chars cause my Curses interface to use ASCII substitutes > for semigraphics. In Unicode-enabled Haskell implementations > (i.e. development versions of ghc) it represents semigraphics > as appropriate Unicode characters, Haskell characters are represented internal to nhc98 as 32-bit values, and always have been. I don't think Unicode should be a particular problem, although as you point out, we don't yet have a library for conversions of char-encodings between 8-bit ASCII, UTF-8, UTF-16, and UTF-32. > [ hmake ] What I want > to say is that sources of these files are in different directories, > and their *.o and *.hi files don't exist yet, but they will be in > the directory where the sources reside. > > Is it possible to let hmake generate dependencies with that assumption? > Or could hmake be changed to support such scheme? Yes, it is possible. I've just committed a small patch to CVS which does what you want. Regards, Malcolm

3 2