- FFI - Haskell.org

RE: PackedString
by Simon Marlow 10 Jan '01

10 Jan '01

> > > Is anybody actually depending on the old `CString'? Or is > > > everybody using `PackedString' instead? > > > > I don't believe anyone is using PackedString, except for Regex which > > arguably should be using CString instead. I'd like to see a simpler > > PackedString which just corresponded to ByteArrays. > > The nhc98 compiler uses PackedString extensively. The interface is > very standard - all of ghc, hbc, and nhc98 support it. Change the > implementation by all means if you wish, but not the interface. Not a problem. I wasn't suggesting changing the interface, sorry for the confusion. Cheers, Simon

1 0

RE: Let's get this finished
by Simon Marlow 10 Jan '01

10 Jan '01

> Is anybody actually depending on the old `CString'? Or is > everybody using `PackedString' instead? I don't believe anyone is using PackedString, except for Regex which arguably should be using CString instead. I'd like to see a simpler PackedString which just corresponded to ByteArrays. Cheers, Simon

3 2

RE: Let's get this finished
by Simon Marlow 10 Jan '01

10 Jan '01

> "Manuel M. T. Chakravarty" wrote: > > If hslibs is meant to be used with systems other than ghc, > > too - which I think was the idea - there is no choice but to > > rewrite it into H98. But I guess this is essentially up to > > Mr. HsLibs aka SimonM. > > Well, doing some simple local transformations to get H98 compliance > should be possible without consulting Mr. HsLibs in advance. :-) [ blimey. I go away for a couple of days and you guys are going bananas. Slow down! ] I should say that I've recently been working on moving most of the FFI code into ghc/lib/std, removing GHC's use of deprecated features (eg. using Ptr instead of Addr), and converting some of GHC's standard libraries to use the new FFI. Directory has been converted so far. I think there's no alternative to having two copies of much of the code - a H98 version in hslibs/lang and GHC-specific code partly in ghc/lib/std and partly in hslibs/lang, with appropriate #ifdefs. We'll need to set up a way to test the H98 code using GHC too. I don't see any problem with using hsc2hs for the H98 versions, since hsc2hs is mostly compiler-independent. (but I notice you've used autoconf instead of hsc2hs for the errno code. I haven't looked at it yet.) Cheers, Simon

2 1

RE: Let's get this finished
by Simon Marlow 10 Jan '01

10 Jan '01

> > [ MarshalError ] > > But SimonM said that he doesn't want to make errno Haskell > > thread local... > > Wasn't this only due to efficiency reasons? IIRC Fergus had a good > suggestion (copying errno only on thread switches or something), so > I'd like to hear Mr. HsLibs' opinion on this topic again. It'll have to be done (copying errno on thread switches, that is). But I still hate it. > > [ MarshalUtils ] > > BTW, can't we just use `with' now. IIRC, the only reason > > Marcin used `withObject' was because of a clash with the > > implicit parameter extension. This has been changed now, > > hasn't it? > > Nope, not yet. Although there was some consensus about SPJ's > ?-suggestion for those bindings, it's not implemented yet. > Any takers? If this has been done, `with' would be better, indeed. Agreed. > > > And just a final topic which really should be settled: > > > "Marshal" or "Marshall"?? :-} > > > > I would have taken the latter, but I think, SimonM > > originally proposed the name MarshalUtils with one "l" and > > as he is the native speaker... dictionary.com recons that 'Marshal' is the correct spelling. 'Marshall' is a military rank only, AFAICT. Cheers, Simon

1 0

Let's get this finished
by Manuel M. T. Chakravarty 10 Jan '01

10 Jan '01

The major piece missing from the FFI library is the high-level marshalling support. Moreover, `MarshalUtils' isn't quite there. In fact, I think, we should have a family of MarshalXXX modules, which include what currently is in MarshalUtils plus some more stuff. MarshalUtils ~~~~~~~~~~~~ The reason that I don't like MarshalUtils as it is are the following: (1) It isn't H98 and in fact its interface cannot be implemented with H98[1]. The problem are the routines with signatures in patterns: malloc :: Storable a => IO (Ptr a) malloc :: IO (Ptr a) = mallocBytes (sizeOf (undefined::a)) alloca :: Storable a => (Ptr a -> IO b) -> IO b alloca (act :: Ptr a -> IO b) = allocaBytes (sizeOf (undefined::a)) act <etc> (2) pokeArray, peekArray, etc are really part of what should go into the high-level marshalling. Point (2) is easily resolved by defining a family of modules including all the necessary functionality. However, Point (1) is a problem. First of all, I guess, we have to revert to the signature for allocation that I originally used in the C2HS library: malloc :: Storable a => a -> IO Addr malloc dummy = mallocBytes (sizeOf dummy) All the other allocation routines have to be altered in a similar way. A similar problem arises for copyArray and moveArray, where I guess the additional argument is in practice more awkward as for the allocation routines. Finally, indexPtr is affected, which is especially annoying since it can't be used infix anymore after the change. The Marshal modules ~~~~~~~~~~~~~~~~~~~ I propose the following set of modules: - MarshalAlloc: The allocation routines that are in MarshalUtils now, but modified as outlined above. - MarshalArray: The array marshalling routines - MarshalString: Most people seem to be of the opinion that we have to handle strings specially due to Unicode conversion requirements. So, this is were string marshalling goes. - MarshalError: Combinators to test for common error conditions. The main open problem here is the handling of errno, I think. - MarshalConv: The C2HS IntConv, BoolConv, etc classes. Marcin just uses `fromIntegral' here, but I think that this is too limited - eg, it doesn't handle Bool. Moreover, soemthing along the C2HS ToAddr/FromAddr classes, but it probably makes to have `stdArrayPtr' etc as additional member functions (this is similar to the newXXX routines that are in MarshalUtils now, but with additional overloading). Furthermore, Marcin-syle withXXX routines would fo here. - MarshalUtils: The memcpy/memmove stuff, indexPtr (if it still makes sense), and whatever else doesn't get its own module. I think, we should leave C2HS's marshX routines out. We discussed how to make them arity independent for quite a while and AFAIR, there wasn't a really good proposal. Finally, all the above modules should be jointly exported by a module `Marshal'. Comments? Cheers, Manuel [1] Which unfortunately escaped me when we discussed MarshalUtils for the first time.

5 27

Re: nhc98-1.{00,01} produce crashing programs
by Malcolm Wallace 08 Jan '01

08 Jan '01

[ Re: FFI in nhc98 - followups set to ffi(a)haskell.org ] > > Int/Word 8/16/64 are not really supported properly at all right now. > > With QForeign they work well enough that the Curses interface works. > QForeign provides Num/Integral/etc. instances, and Int64/Word64 are > not used much. Yesterday, I commited lots of code for Int/Word 8/16/32/64. Some work is still required for 64-bit value conversions only. > I miss foreign export dynamic the most. This is the trickiest part to implement, of course. As far as I know, we can lift the most difficult sections directly out of Hugs or ghc's implementation and re-use it. (But see further comment about foreign import dynamic below.) > Then the fact that case does > not work for arbitrary numeric types. I don't understand - can you explain? Surely the Num instances for Word/Int 8/16/32/64 mean that a literal integer 3 in the source code becomes (fromInteger 3) in reality, and this should be fine in case comparisons. > Then foreign label and foreign import dynamic. Foreign label is probably not too difficult to get going quickly. As to dynamic import and export, I have never had any luck playing with dynamic loading of libraries in C. Can anyone recommend a good tutorial? The manual pages I have read so far were not very enlightening. I'm afraid that, until I understand how to get dynamic libraries going in C, there will be no provision for them in nhc98. > addForeignPtrFinalizer is not emulated under nhc because it does not > provide addForeignFinalizer :: ForeignObj -> IO () -> IO () > (this function is not so important). Specifying a finalizer as IO () (i.e. as Haskell code rather than C code) is not a straightforward idea at all. Currently in nhc98, such finalizers are recorded, but may never be run. This will be fixed eventually. However, as far as I am aware, we agreed to remove the idea of *multiple* finalizers from the common FFI spec, because they have too many potential semantic problems. > QForeign comes with part-of-Parsec and GetOpt from ghc, which are > needed by it but not distributed with nhc. The next release of nhc98 will include GetOpt, Parsec, and a whole host of other libraries. > It also emulates ghc's Bits module with Bits class; nhc's Bit is different. Yes, nhc98's Bit library follows version 1.3 of the Haskell Library Report, whereas ghc's Bits library has never been standard. :-) Fortunately, the next release of nhc98 can provide both interfaces if you wish. > nhc does not provide CTypes and CTypesISO modules, nor HsFFI.h CTypes and HsFFI.h exist as of yesterday. CTypesISO will follow shortly. > 8-bit Chars cause my Curses interface to use ASCII substitutes > for semigraphics. In Unicode-enabled Haskell implementations > (i.e. development versions of ghc) it represents semigraphics > as appropriate Unicode characters, Haskell characters are represented internal to nhc98 as 32-bit values, and always have been. I don't think Unicode should be a particular problem, although as you point out, we don't yet have a library for conversions of char-encodings between 8-bit ASCII, UTF-8, UTF-16, and UTF-32. > [ hmake ] What I want > to say is that sources of these files are in different directories, > and their *.o and *.hi files don't exist yet, but they will be in > the directory where the sources reside. > > Is it possible to let hmake generate dependencies with that assumption? > Or could hmake be changed to support such scheme? Yes, it is possible. I've just committed a small patch to CVS which does what you want. Regards, Malcolm

3 2

RE: Let's get this finished
by Simon Marlow 07 Jan '01

07 Jan '01

> MarshalUtils > ~~~~~~~~~~~~ > The reason that I don't like MarshalUtils as it is are the > following: > > (1) It isn't H98 and in fact its interface cannot be > implemented with H98[1]. The problem are the routines > with signatures in patterns: > > malloc :: Storable a => IO (Ptr a) > malloc :: IO (Ptr a) = mallocBytes (sizeOf (undefined::a)) > > alloca :: Storable a => (Ptr a -> IO b) -> IO b > alloca (act :: Ptr a -> IO b) = allocaBytes (sizeOf > (undefined::a)) act > > <etc> How about this: malloc :: Storable a => IO (Ptr a) malloc = res where res = mallocBytes (sizeOf (unsafePerformIO (res>>=peek))) :-) The result type signatures are just a convenient (and much more readable) way to write these functions. > (2) pokeArray, peekArray, etc are really part of what should > go into the high-level marshalling. Ok. Actually I consider the current libraries to be on the verge of usability, modulo the missing string marshalling and error functionality. > The Marshal modules > ~~~~~~~~~~~~~~~~~~~ > I propose the following set of modules: > > - MarshalAlloc: > The allocation routines that are in MarshalUtils now, > but modified as outlined above. > > - MarshalArray: > The array marshalling routines > > - MarshalString: > Most people seem to be of the opinion that we have to > handle strings specially due to Unicode conversion > requirements. So, this is were string marshalling goes. I've been wondering whether we should have something like newtype CString = CString (Ptr CChar) which would allow GHC to replace the representation with a ByteArray leading to a much more efficient implementation of withCString. There are other reasons why having a special CString type might be a good idea: a C string really is more than just a pointer to char, it is an array with a zero terminator and is specified in some kind of encoding. Having a special CString type would provide more type safety. One problem with the optimisation is that GHC can't handle ByteArrays being *returned* from a foreign function. > - MarshalError: > Combinators to test for common error conditions. The > main open problem here is the handling of errno, I > think. > > - MarshalConv: > The C2HS IntConv, BoolConv, etc classes. Marcin just > uses `fromIntegral' here, but I think that this is too > limited - eg, it doesn't handle Bool. Could you explain the motivation for these? I can see the need for some way to convert Bool<=>CInt (which is defined by ISO C), but can't the others be just fromIntegral? > Moreover, > soemthing along the C2HS ToAddr/FromAddr classes, but it > probably makes to have `stdArrayPtr' etc as additional > member functions (this is similar to the newXXX routines > that are in MarshalUtils now, but with additional > overloading). Furthermore, Marcin-syle withXXX routines > would fo here. I'm not sure how ToAddr/FromAddr would work. From looking at C2HS, they seem to provide a consistent interface to marshalling that is specific to a given Haskell type. I haven't found a need for something like this in my own code yet, so again could you motivate? > - MarshalUtils: > The memcpy/memmove stuff, indexPtr (if it still makes > sense), and whatever else doesn't get its own module. > > I think, we should leave C2HS's marshX routines out. We > discussed how to make them arity independent for quite a > while and AFAIR, there wasn't a really good proposal. > Finally, all the above modules should be jointly exported by > a module `Marshal'. Sounds fine. Cheers, Simon

5 10

RE: qforeign-0.62
by Simon Marlow 04 Jan '01

04 Jan '01

> qrczak(a)knm.org.pl (Marcin 'Qrczak' Kowalczyk) wrote, > > > qforeign got smaller because of changes in ghc, but what is ghc is > > still a bit incomplete. > > > > Most important incompleteness is in string handling. Since ghc's > > Unicode support is half-baked and it is not yet clear how > the library > > interface will look like, I propose adding the following functions > > to the module Foreign: > > > > peekCString :: Ptr CChar -> IO String > > peekCStringLen :: Ptr CChar -> Int -> IO String > > > > withCString :: String -> (Ptr CChar -> IO a) -> IO a > > withCStringLen :: String -> (Ptr CChar -> Int -> IO a) -> IO a > > > > newCString :: String -> IO (Ptr CChar) > > newCStringLen :: String -> IO (Ptr CChar, Int) > > -- Memory is obtained by malloc and must be freed using free. > > I believe that this stuff belongs into the higher-level > marshalling library. In fact, it should be instances of > more general routines dealing with lists of Storable > elements. Similar to how it is done in the C2HS library. I think Marcin is right in that C Strings are special (unfortunately). This interface seems reasonable, and simple. Do we have any objections? > > ---------------------------------------------------------------- > > -- Storable arrays. No new expressiveness, but they are commonly > > -- used. > > > > peekArray :: Storable a => Ptr a -> Int -> IO [a] > > peekArray0 :: (Storable a, Eq a) => a -> Ptr a -> IO [a] > > > > pokeArray :: Storable a => Ptr a -> [a] -> IO () > > pokeArray0 :: Storable a => a -> Ptr a -> [a] -> IO () > > > > withObject :: Storable a => a -> (Ptr a -> IO b) -> IO b > > withArray :: Storable a => [a] -> (Ptr a -> IO b) -> IO b > > withArray0 :: Storable a => a -> [a] -> (Ptr a -> IO b) -> IO b > > > > newObject :: Storable a => a -> IO (Ptr a) > > newArray :: Storable a => [a] -> IO (Ptr a) > > newArray0 :: Storable a => a -> [a] -> IO (Ptr a) > > -- Memory is obtained by malloc and must be freed using free. > > This seems to go into the direction that I mentioned above. Ok. The naming scheme is a little inconsistent: we haven't used "Object" everywhere (it is just omitted in some places). eg. malloc :: Storable a => IO (Ptr a) arguably, withObject and newObject should be named with and new ("Object" is awfully vague and seems redundant anyway). > > ---------------------------------------------------------------- > > -- Handling errors. Mostly conversion of errno to Haskell's IO > > -- exceptions. > > > > -- Get the current value of C errno. > > getErrno :: IO Int > > > > -- Throw an exception based on function name and given > error code. > > throwCError :: String -> Int -> IO a > > -- Throw an exception based on function name and current errno > > -- value. Much more common than the above but less general, > > -- that's why there are two functions. > > throwErrno :: String -> IO a > > > > -- Perform an action and ignore its result. A stupid > thing but quite > > -- common, especially after the result was converted to > exceptions. > > ignore :: IO a -> IO () > > > > -- Wrappers that convert the result to a possible exception. > > -- *_EINTR variants repeat the action if there was an error > > -- and errno == EINTR. > > throwIf :: (a -> Bool) -> String -> IO a -> IO a > > throwIf_EINTR :: (a -> Bool) -> String -> IO a -> IO a > > > > throwIfMinus1 :: Num a => String -> IO a -> IO a > > throwIfMinus1_EINTR :: Num a => String -> IO a -> IO a > > throwIfNull :: String -> IO (Ptr a) -> IO (Ptr a) > > throwIfNull_EINTR :: String -> IO (Ptr a) -> IO (Ptr a) > > Again, this belongs into the higher-level marshalling > library. Did you have a look at the corresponding routines > in C2HS? Your ifRaise family is fairly similar (except that it doesn't deal explicitly with C's errno and EINTR). I don't particularly mind what these functions are called or where they live, as long as we have them! Admittedly the functions above are C-specific and should probably go into a C-specific library on top of Foreign. CForeign anyone? Meanwhile we should have some non-C-specific error-handling utilities in Foreign. Cheers, Simon

8 36

RE: freeHaskellFunctionPtr
by Simon Marlow 04 Jan '01

04 Jan '01

> Tue, 02 Jan 2001 19:40:11 +1100, Manuel M. T. Chakravarty > <chak(a)cse.unsw.edu.au> pisze: > > > PS: It could be argued that `Ptr ()' can always be used > > instead of `Addr' and that therefore we should do away > > with `Addr' altogether. Opinions? > > I agree that Addr uses should be replaced by Ptr and FunPtr > (well, I thought it was obvious!). Yup, I agree with all this. We should deprecate Addr in the interests of having as few interfaces as possible. > newForeignObj has been converted to Ptr, but since ForeignObjs > themselves were later replaced with ForeignPtrs, it makes sense to > let newForeignObj use Addr again as in previous released versions of > ghc. For withForeignObj it could be done for consistency too (it's > new so it does not really matter). Sounds reasonable. > I would let freeHaskellFunPtr accept and ignore nullFunPtr, since I > once had to write > when (ptr /= nullFunPtr) $ freeHaskellFunPtr ptr > and it can be seen as consistent with free. This makes sense if nullFunPtr is really a valid value for a FunPtr. The FFI spec doesn't say anything about f.e.d. functions possibly returning nullFunPtr. I've always thought that C's free() accepting NULL as a valid argument was a questionable decision (although I have to admit it's convenient at times). Cheers, Simon

6 12

freeHaskellFunctionPtr
by Manuel M. T. Chakravarty 02 Jan '01

02 Jan '01

I propose to deprecate `freeHaskellFunctionPtr' in Foreign. Equivalent functionality is now available from `Ptr.freeHaskellFunPtr'. Normally, we have all `Ptr' functionality also in a variant for `Addr', but in this case, this doesn't make sense in my opinion. The reason is that `Addr' is there as a backup solution when `Ptr' doesn't fit. But if a `Addr' is used to represent a Haskell function pointer, I don't see a reason that `Ptr.FunPtr' could not have been taken. If there are no objections, I will deprecate `freeHaskellFunctionPtr' in `Foreign'. Cheers, Manuel PS: It could be argued that `Ptr ()' can always be used instead of `Addr' and that therefore we should do away with `Addr' altogether. Opinions?

3 2