qrczak@knm.org.pl (Marcin 'Qrczak' Kowalczyk) wrote,
qforeign got smaller because of changes in ghc, but what is ghc is still a bit incomplete.
Most important incompleteness is in string handling. Since ghc's Unicode support is half-baked and it is not yet clear how the library interface will look like, I propose adding the following functions to the module Foreign:
peekCString :: Ptr CChar -> IO String peekCStringLen :: Ptr CChar -> Int -> IO String
withCString :: String -> (Ptr CChar -> IO a) -> IO a withCStringLen :: String -> (Ptr CChar -> Int -> IO a) -> IO a
newCString :: String -> IO (Ptr CChar) newCStringLen :: String -> IO (Ptr CChar, Int) -- Memory is obtained by malloc and must be freed using free.
I believe that this stuff belongs into the higher-level marshalling library. In fact, it should be instances of more general routines dealing with lists of Storable elements. Similar to how it is done in the C2HS library.
I think Marcin is right in that C Strings are special (unfortunately). This interface seems reasonable, and simple. Do we have any objections?
---------------------------------------------------------------- -- Storable arrays. No new expressiveness, but they are commonly -- used.
peekArray :: Storable a => Ptr a -> Int -> IO [a] peekArray0 :: (Storable a, Eq a) => a -> Ptr a -> IO [a]
pokeArray :: Storable a => Ptr a -> [a] -> IO () pokeArray0 :: Storable a => a -> Ptr a -> [a] -> IO ()
withObject :: Storable a => a -> (Ptr a -> IO b) -> IO b withArray :: Storable a => [a] -> (Ptr a -> IO b) -> IO b withArray0 :: Storable a => a -> [a] -> (Ptr a -> IO b) -> IO b
newObject :: Storable a => a -> IO (Ptr a) newArray :: Storable a => [a] -> IO (Ptr a) newArray0 :: Storable a => a -> [a] -> IO (Ptr a) -- Memory is obtained by malloc and must be freed using free.
This seems to go into the direction that I mentioned above.
Ok. The naming scheme is a little inconsistent: we haven't used "Object" everywhere (it is just omitted in some places). eg. malloc :: Storable a => IO (Ptr a) arguably, withObject and newObject should be named with and new ("Object" is awfully vague and seems redundant anyway).
---------------------------------------------------------------- -- Handling errors. Mostly conversion of errno to Haskell's IO -- exceptions.
-- Get the current value of C errno. getErrno :: IO Int
-- Throw an exception based on function name and given
error code.
throwCError :: String -> Int -> IO a -- Throw an exception based on function name and current errno -- value. Much more common than the above but less general, -- that's why there are two functions. throwErrno :: String -> IO a
-- Perform an action and ignore its result. A stupid
thing but quite
-- common, especially after the result was converted to
exceptions.
ignore :: IO a -> IO ()
-- Wrappers that convert the result to a possible exception. -- *_EINTR variants repeat the action if there was an error -- and errno == EINTR. throwIf :: (a -> Bool) -> String -> IO a -> IO a throwIf_EINTR :: (a -> Bool) -> String -> IO a -> IO a
throwIfMinus1 :: Num a => String -> IO a -> IO a throwIfMinus1_EINTR :: Num a => String -> IO a -> IO a throwIfNull :: String -> IO (Ptr a) -> IO (Ptr a) throwIfNull_EINTR :: String -> IO (Ptr a) -> IO (Ptr a)
Again, this belongs into the higher-level marshalling library. Did you have a look at the corresponding routines in C2HS?
Your ifRaise family is fairly similar (except that it doesn't deal explicitly with C's errno and EINTR). I don't particularly mind what these functions are called or where they live, as long as we have them! Admittedly the functions above are C-specific and should probably go into a C-specific library on top of Foreign. CForeign anyone? Meanwhile we should have some non-C-specific error-handling utilities in Foreign. Cheers, Simon
Marcin proposes adding the following functions to the module Foreign:
peekCString :: Ptr CChar -> IO String peekCStringLen :: Ptr CChar -> Int -> IO String
withCString :: String -> (Ptr CChar -> IO a) -> IO a withCStringLen :: String -> (Ptr CChar -> Int -> IO a) -> IO a
newCString :: String -> IO (Ptr CChar) newCStringLen :: String -> IO (Ptr CChar, Int)
Manuel says
I believe that this stuff belongs into the higher-level marshalling library.
Simon says
I think Marcin is right in that C Strings are special (unfortunately). This interface seems reasonable, and simple. Do we have any objections?
I'm with Manuel. Module FFI/Foreign is already stable, and I object to adding things to it arbitrarily. By all means make a new module with these functions, which I can see could be useful. But please don't just allow "feature creep" over already-agreed specifications. Once these newer features are settled, I look forward to seeing a portable implementation.
---------------------------------------------------------------- -- Handling errors. Mostly conversion of errno to Haskell's IO -- exceptions.
-- Get the current value of C errno. getErrno :: IO Int
-- Throw an exception based on function name and given error code. throwCError :: String -> Int -> IO a -- Throw an exception based on function name and current errno -- value. Much more common than the above but less general, -- that's why there are two functions. throwErrno :: String -> IO a
Getting hold of error codes from C is a big hole in the current FFI that I agree needs to be plugged. Here's my contribution. I think we should use symbolic error codes in Haskell, like the `ENOENT' style in C. This is for portability, because as you may know, on different operating systems, symbolic names map to different ints. I would hate to see magic numbers like '35' on IRIX but '54' on SunOS appearing in code. In nhc98, we have an enumerated type data ErrNo = EPERM | ENOENT | ESRCH | EINTR | EIO | ENXIO | E2BIG | ENOEXEC | EBADF | ECHILD | .... deriving (Eq,Enum,Show) For some machines Eq is insufficiently strong, and the following kind of patch-up is required: eqErrNo :: ErrNo -> ErrNo -> Bool eqErrNo EWOULDBLOCK EAGAIN = True eqErrNo EAGAIN EWOULDBLOCK = True eqErrNo EDEADLOCK EDEADLK = True eqErrNo EDEADLK EDEADLOCK = True eqErrNo ENOTSUP EOPNOTSUPP = True eqErrNo EOPNOTSUPP ENOTSUP = True eqErrNo a b = a==b On other machines, derived Enum is not accurate either, due to discontinuities in the range, and as noted earlier, symbolic names can map to different ints. Finally, each individual operating system only actually recognises a (differing) subset of all the symbols that are possible. For these reasons, we need to generate the ErrNo module automatically for any given machine (auto-generator available on request), but the benefit is better code portability in Haskell. So I would suggest a refinement to Marcin's interface as follows:
getErrno :: IO ErrNo
implemented as (do i <- primGetErrNo return (toEnum i)) and a corresponding:
throwCError :: String -> ErrNo -> IO a
Regards, Malcolm
On Tue, 28 Nov 2000 malcolm-ffi@cs.york.ac.uk wrote:
I'm with Manuel. Module FFI/Foreign is already stable, and I object to adding things to it arbitrarily. By all means make a new module with these functions, which I can see could be useful. But please don't just allow "feature creep" over already-agreed specifications.
C string functions are necessary to exchange strings with C. Ok, they may be put in CForeign, but without a common interface that will at some point switch to proper charset handling it's impossible to correctly interface to most C libraries.
Once these newer features are settled, I look forward to seeing a portable implementation.
I don't understand. How can they settle if they are absent? And most of discussed functions (except C strings and errno) do have portable implementations in terms of other functions. I see no point in waiting forever until the thin layer of supplementary convenience wrappers for array handling is established - it will not appear if we don't make it.
I think we should use symbolic error codes in Haskell, like the `ENOENT' style in C. This is for portability, because as you may know, on different operating systems, symbolic names map to different ints. I would hate to see magic numbers like '35' on IRIX but '54' on SunOS appearing in code.
First of all, in the case of errno I don't see the need of exposing particular constans to Haskell programmers. Usually it's enough (and necessary) to convert errno to Haskell's exception. There are sometimes cases where specific errno values are needed. The situation is no different than for other symbolic constants, like F_GETFL. There is just a need for a more convenient generic solution than wrapping all these values in functions - which is portable but really painful. It does not matter if it will be hsc2hs or something different, but it will be needed anyway, so we don't have to go a hard way for errno.
data ErrNo = EPERM | ENOENT | ESRCH | EINTR | EIO | ENXIO | E2BIG | ENOEXEC | EBADF | ECHILD | .... deriving (Eq,Enum,Show)
I think it's a bad idea. Not only for errno, but for many similar things, e.g. socket types in module SocketPrim. It's not extensible. Assume that one standard defines some constants and another includes the previous one but defines more. Then in your framework the very first implementation must know all values from all standards, hide ones that don't belong to the standard it exposes, export them through a private module and let other modules reexport them for the wide audience. It really bites in practice. Linux kernels 2.2.x provide packet sockets. I cannot use them with ghc's Socket / SocketPrim because the set of supported enums is hardwired into the implementation! I propose one of the following methods for such cases. First is to use a non-abstract type - either a type like CInt which directly corresponds to these values in C, or a newtype with the exported constructor, or a type with Num instance. Export values known to be present as plain variables. Then a programmer which must access a value which was not supported can do it. The second way is paranoid: do the same but export the type abstractly. This is as safe as your solution, but uses the same style of interface as non-paranoid solutions, and allows exporting the representation through a private module for those who really need it. Both solutions are better than algebraic types wrt. efficiency and the ease of implementation. Efficiency is obvious: values need not to be remapped each time. And it's very easy to implement it when we have some generic mechanism for accessing enums and #defines from Haskell, like hsc2hs. I tried to go the algebraic type way without hsc2hs - it is very painful and requires a third layer between. Also there is no problem what to do with error codes we don't understand. Just pass them unmodified to the user of the module - he must handle some default anyway. Since there is currently no way to let the syntax of constructors represent an abstract datatype ("views"), algebraic types should not be abused. They are applicable when the set of constants is really decided once and forever. My interface can be a wrapper for your implementation but it's harder in the opposite direction.
For some machines Eq is insufficiently strong, and the following kind of patch-up is required:
eqErrNo :: ErrNo -> ErrNo -> Bool eqErrNo EWOULDBLOCK EAGAIN = True eqErrNo EAGAIN EWOULDBLOCK = True eqErrNo EDEADLOCK EDEADLK = True eqErrNo EDEADLK EDEADLOCK = True eqErrNo ENOTSUP EOPNOTSUPP = True eqErrNo EOPNOTSUPP ENOTSUP = True eqErrNo a b = a==b
Not needed in my solution. You can export the same constant under several names. -- Marcin 'Qrczak' Kowalczyk
Once these newer features are settled, I look forward to seeing a portable implementation.
I don't understand. How can they settle if they are absent?
What do you mean by "absent"? I thought you had a proposal. I was just asking for it to be recognised as a proposal rather than a standard, because we haven't had any experience with using it yet. Just by looking at the names and type signatures, I'm not even sure what each proposed function does. Could you describe them in a little more detail? My request for a portable implementation is because, now that the basic common FFI is settled, I believe we should be using it to ensure that new proposals such as yours are not restricted to GHC-only, like so many existing libraries sadly are.
And most of discussed functions (except C strings and errno) do have portable implementations in terms of other functions.
Ah, are you saying that treatment of C strings cannot have an implementation that is portable across Haskell compilers? I'm sorry, but if this was your intention, it wasn't clear from your original message. Regards, Malcolm
On Tue, 28 Nov 2000 malcolm-ffi@cs.york.ac.uk wrote:
What do you mean by "absent"? I thought you had a proposal. I was just asking for it to be recognised as a proposal rather than a standard, because we haven't had any experience with using it yet.
Ok.
Just by looking at the names and type signatures, I'm not even sure what each proposed function does. Could you describe them in a little more detail?
Sorry, I thought it was clear from names and types. peekCString :: Ptr CChar -> IO String -- Read a null-terminated string of C chars from memory and convert -- it to Haskell string. The C string is assumed to be in the default -- local byte encoding. peekCStringLen :: Ptr CChar -> Int -> IO String -- Similarly, but read the specified number of chars instad of searching -- for '\0'. withCString :: String -> (Ptr CChar -> IO a) -> IO a -- Convert a Haskell string to the null-terminated C string in the -- default local byte encoding, stored in some temporary memory. -- Apply the function to the pointer to that string, execute the -- resulting action and then free the string. withCStringLen :: String -> (Ptr CChar -> Int -> IO a) -> IO a -- Similarly, but tells about the length of the converted string, -- not including the final '\0'. This allows converting strings -- containing null characters. newCString :: String -> IO (Ptr CChar) newCStringLen :: String -> IO (Ptr CChar, Int) -- Like withCString*, but the memory is obtained by malloc. It must be -- explicitly freed using free.
My request for a portable implementation is because, now that the basic common FFI is settled, I believe we should be using it to ensure that new proposals such as yours are not restricted to GHC-only, like so many existing libraries sadly are.
The C string stuff can be written portably in Haskell implementations which use the same character encoding as C instead of Unicode. That implementation will not be valid when Strings really use Unicode. The generic interface for charset conversions is not established yet. I have a proposal for it, but there is much room for choice there. Having such interface it is possible to write the C string functions in an inefficient way. My proposal for charsets allows having more efficient C string functions than can be written using its official interface (avoiding going through Haskell lists of CChars or Bytes). In practice the implementations of Unicode stuff and C string stuff are mutually dependent. -- Marcin 'Qrczak' Kowalczyk
"Marcin 'Qrczak' Kowalczyk" <mk167280@zodiac.mimuw.edu.pl> wrote,
On Tue, 28 Nov 2000 malcolm-ffi@cs.york.ac.uk wrote:
I think we should use symbolic error codes in Haskell, like the `ENOENT' style in C. This is for portability, because as you may know, on different operating systems, symbolic names map to different ints. I would hate to see magic numbers like '35' on IRIX but '54' on SunOS appearing in code.
First of all, in the case of errno I don't see the need of exposing particular constans to Haskell programmers. Usually it's enough (and necessary) to convert errno to Haskell's exception.
I understand that Malcolm wants to be able to get to the constants, but
There are sometimes cases where specific errno values are needed. The situation is no different than for other symbolic constants, like F_GETFL. There is just a need for a more convenient generic solution than wrapping all these values in functions - which is portable but really painful. It does not matter if it will be hsc2hs or something different, but it will be needed anyway, so we don't have to go a hard way for errno.
I also agree with this. Hardcoded data types should be avoided were possible.
I propose one of the following methods for such cases. First is to use a non-abstract type - either a type like CInt which directly corresponds to these values in C, or a newtype with the exported constructor, or a type with Num instance. Export values known to be present as plain variables. Then a programmer which must access a value which was not supported can do it.
This sounds reasonable to me.
The second way is paranoid: do the same but export the type abstractly. This is as safe as your solution, but uses the same style of interface as non-paranoid solutions, and allows exporting the representation through a private module for those who really need it.
I don't really fully understand what you want to say here... Cheers, Manuel
Fri, 01 Dec 2000 12:53:26 +1100, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
The second way is paranoid: do the same but export the type abstractly. This is as safe as your solution, but uses the same style of interface as non-paranoid solutions, and allows exporting the representation through a private module for those who really need it.
I don't really fully understand what you want to say here...
Forget it... When somebody feels bad allowing calling his functions with bogus or non-portable arguments, and wants to provide the type abstractly, then he can do it in the "variable" scheme too. The scheme can be extended to allow retrospective adding of new constants by splitting the module into two, like Socket + SocketPrim, and exporting the type non-abstractly only from the lower layer. Thus we have both a nice abstract interface and a way to fiddle with non-standard constants. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
I understand that Malcolm wants to be able to get to the constants, but
There are sometimes cases where specific errno values are needed. The situation is no different than for other symbolic constants, like F_GETFL.
I also agree with this. Hardcoded data types should be avoided were possible.
Yes, I am persuaded that hard-coded datatypes tend to be bad in this situation. The way hsc2hs deals with constants is quite good, and it is certainly a better, more flexible, approach. However, I do have some hesitation in relying on yet another pre-processing tool, with yet another "little language" outside of Haskell itself. Of course hsc2hs does more than just filling in constants, but for just that one task, I wonder if we should be extending the primitive FFI mechanism itself. For instance, we already have "foreign label", so what about perhaps "foreign value": foreign value "F_GETFL" fGetFL :: CInt This is just off the top of my head, so I'm probably missing some subtleties. What do people think of this idea? Are there some real killer situations which would make it impossible? Regards, Malcolm
Fri, 1 Dec 2000 14:45:32 +0000, malcolm-ffi@cs.york.ac.uk <malcolm-ffi@cs.york.ac.uk> pisze:
Of course hsc2hs does more than just filling in constants, but for just that one task, I wonder if we should be extending the primitive FFI mechanism itself.
I once proposed something which is related to ghc's _casm_ as foreign import is related to _ccall_. AFAIR nobody seemed interested. C macros with parameters are harder than constants. What is worse: sometimes it is not clearly documented whether a function provided by a library is really a function or can be also a macro. There is a fundamental difference between functions and variables on one side, and macros, enum constants and struct fields on the other. Using objects of the first kind does not require cooperation with a C compiler. Everything needed to use the object are determined by the official interface. A programmer writes a foreign declaration which matches the documentation and it's fine. OTOH the details for the second kind are hidden in C headers and may differ between versions of the library. Current foreign declarations support only the first kind. They are implementable in a native code generator. They are even independent of the language, provided that names are not mangled, types are compatible and the calling convention is supported. Approaching the second kind in the core language is tricky. First of all, it must be somehow defined which C headers to include to find a definition, and what preprocessor symbols to #define to make it available. Don't forget about specifying which C compiler to use and with which options, e.g. which include paths. The whole thing is more language specific than foreign declarations for the first kind. There are two separate issues: 1. Is it possible to have the second kind from the level of the Haskell compiler at all (done in an sufficiently elegant way to think about it as a candidate proposal for the official FFI)? 2. If so, which features are worth supporting? Perhaps we could completely get rid of an equivalent of hsc2hs. We could dream about fantastic features for the second question, but it's worth nothing if the answer to the first one is NO. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
On 01-Dec-2000, Marcin 'Qrczak' Kowalczyk <qrczak@knm.org.pl> wrote:
There is a fundamental difference between functions and variables on one side, and macros, enum constants and struct fields on the other.
Using objects of the first kind does not require cooperation with a C compiler. ... Current foreign declarations support only the first kind. They are implementable in a native code generator. They are even independent of the language, provided that names are not mangled, types are compatible and the calling convention is supported.
Approaching the second kind in the core language is tricky. First of all, it must be somehow defined which C headers to include to find a definition, and what preprocessor symbols to #define to make it available.
Mercury supports this using `foreign_decls' and `foreign_code' pragmas. Originally we just supported C, and so these were named `c_header_code' and `c_code' respectively, but we're currently in the process of generalizing this to support arbitrary languages. The `foreign_decls' pragma lets you include any foreign language declarations that the foreign language code might need, e.g. :- pragma foreign_decls(c, " #define _POSIX_SOURCE 1 #include "conf.h" #ifdef HAVE_UNISTD_H #include <unistd.h> #else #include <stdio.h> #endif extern int my_global; "). For the `foreign_code' pragma, there are two forms. The first form just specifies a block of foreign code that will be compiled with the foreign language compiler and linked into the program. E.g. :- pragma foreign_code(c, " int my_global; "). This form is no strictly necessary, since you could always write such code in the foreign language itself, rather than in Mercury code inside `pragma foreign_code', but it is often very convenient. The other form is for when you are implementing a Mercury procedure using foreign code. You give the declaration of the Mercury procedure, e.g. :- func sin(float) = float. :- mode sin(in) = out is det. % this line is optional and then you give a `pragma foreign_code' declaration to implement it: :- pragma foreign_code( % specify which language c, % specify which function this is for, % and which mode of the function, and % also introduce variable names for the arguments sin(X::in) = (Y::out), % a list of flags; these can give the Mercury % compiler additional information to help it % generate more efficient code [will_not_call_mercury, thread_safe], % The foreign language code to implement the procedure % This uses the variable names introduced earlier "Y = sin(X);" ). For the Mercury -> MS IL back-end (and soon for the Mercury -> Java back-end), the Mercury compiler extracts the foreign_decl and foreign_code stuff, wrapping code in the second form of foreign_code up inside appropriate function definitions, and puts them all in a separate file that it then invokes the foreign language compiler on. This does require a bit of language-specific knowledge in the Mercury compiler, because it needs to know the foreign language syntax for function definitions, so that it can wrap up the code from a `foreign_code' declaration in an externally visible function definition. Currently we only support C, but adding support for other languages would not be hard. For example, for Haskell I suppose if you wrote :- module example. ... :- pragma foreign_decl(haskell, " import Prelude -- this is not a good example ;-) "). :- func haskell_sin(float) = float. :- pragma foreign_code(haskell, [], haskell_sin(x::in) = (y::out), "y = sin x" ). we would generate a Haskell file containing something like this: import Prelude -- this is not a good example ;-) example__haskell_sin_1_f_0 x = y where y = sin x The name `example__haskell_sin_1_f_0' is a mangled name corresponding to the name of the Mercury procedure that this code is for; the "1_f_0" bit means that this is mode number zero of a function with one argument.
Don't forget about specifying which C compiler to use and with which options, e.g. which include paths.
Those are not part of the language. Those are specified by options which you pass to the Mercury compiler, or more likely put in your makefile. E.g. to compile the C example you could use mmc --cc gcc --cflags "-I/foo/bar" example.m (this is currently supported) and to compile the Haskell one you could use mmc --haskell-compiler ghc --haskell-flags \ "-fallow-undecidable-instances" example.m Or you could put both examples in the same module and pass both sets of flags to mmc.
There are two separate issues:
1. Is it possible to have the second kind from the level of the Haskell compiler at all (done in an sufficiently elegant way to think about it as a candidate proposal for the official FFI)?
I think so. I certainly consider the mechanism that Mercury uses to be elegant enough to be the official Mercury FFI, and I hope our standards are not so much lower than yours ;-)
2. If so, which features are worth supporting?
See above. Cheers, Fergus. -- Fergus Henderson <fjh@cs.mu.oz.au> | "I have always known that the pursuit | of excellence is a lethal habit" WWW: <http://www.cs.mu.oz.au/~fjh> | -- the last words of T. S. Garp.
Sat, 2 Dec 2000 13:13:08 +1100, Fergus Henderson <fjh@cs.mu.oz.au> pisze:
1. Is it possible to have the second kind from the level of the Haskell compiler at all (done in an sufficiently elegant way to think about it as a candidate proposal for the official FFI)?
I think so.
I hope that improvement there is possible, because even importing plain C functions to ghc is not perfect: - When using the native code generator, there is no type checking across languages at all. - When compiling via C, the C function sometimes gets inlined in the interface file and called directly from other modules which did not include the C header. It results in a warning about implicit function definition (which is illegal in C99 BTW) and will not work at all for foreign labels. So we don't really have a choice: there should be some notion of a "C preprocessor and declaration environment" needed by a Haskell module. It must be robust enough to make it possible to automatically reproduce the environment for other modules which use the module which introduces the environment. Maybe it should not be transferred unnecessarily. When no C function is inlined in an exported Haskell function, it does not buy anything to transfer the C environment, and there is always the danger of conflicts with other C headers. One minor but annoying thing: when a function is declared in a C header only when a given preprocessor symbol is defined, and the header gets included automatically by ghc's rts, it is too late if the symbol is defined e.g. in another C header which is included using ghc's -#include option. It must be passed to ghc command line as e.g. -optc-D_XOPEN_SOURCE. I'm afraid that such glitches may become a pain if the machinery is automatized in a wrong way. It will surely get worse when aiming at other languages :-) -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
Fergus writes, triggered by my suggestion of "foreign value":
Mercury supports this using `foreign_decls' and `foreign_code' pragmas.
I refrained from proposing this in an earlier mail, but I've been thinking about it for a long time. I *really* *really* want to be able to insert small snippets of foreign code into the source of a Haskell module. GreenCard allows you to do this, via the %- or %C directives. It is one of the features I miss most about the new FFI. I know that it is always possible to write a small .c file and compile and link it in separately, but with the new FFI that now seems to necessitate writing an additional .h file as well. And all for a handful of lines of foreign code that could easily be kept in the Haskell source, with the ease-of-comprehension benefits entailed by keeping everything together that belongs together. For instance, in the new FFI I often want to do something like foreign code { #include <dirent.h> } foreign import opendir :: Ptr String -> IO (Ptr DIR) Ok, you might argue that ghc already lets you do #include's. But it isn't part of the agreed FFI, and other compilers don't have that feature. And often I want to do just a tiny bit more than a simple #include. foreign code { #include <string.h> } foreign import readdirent :: Ptr DIR -> IO (Ptr String) foreign code C { char *readdirent (DIR*d) { struct dirent *e; e = readdir(d); return e->d_name; } } I propose that foreign code <language-id> { <code> } should simply cause the verbatim copying of the <code> part into a file to be compiled by the appropriate foreign compiler. That's all. Nothing fancy. The facility I propose is very similar to what Fergus described in Mercury, but I came up with it independently. As people have pointed out, we need to avoid being totally C-centric, which is why there needs to be a language identifier attached. We can argue over exact syntax if you like, but I think the facility is desperately needed.
:- pragma foreign_decls(c, " #define _POSIX_SOURCE 1 extern int my_global; ").
Fergus, could you enlighten me some more about the difference between the foreign_decls and foreign_code pragmas? I'm afraid I can't immediately see why you have both.
the Mercury compiler extracts the foreign_decl and foreign_code stuff, wrapping code in the second form of foreign_code up inside appropriate function definitions, and puts them all in a separate file that it then invokes the foreign language compiler on.
And I would expect a Haskell compiler to do something similar. For C, Java, and the like, this would be fairly easy for the compiler writer to provide. But in the general case, I'd guess that there should be some sort of configurable "plugin" interface, by which an ordinary user could tell the the Haskell compiler about foreign language name: Wibble compiler name: wblc default options: -fno-flubble-mangling extra linking options: -lwibblelib and such like, so that they could use the FFI for whatever language they pleased.
Don't forget about specifying which C compiler to use and with which options, e.g. which include paths.
Those are not part of the language. Those are specified by options which you pass to the Mercury compiler, or more likely put in your makefile. E.g. to compile the C example you could use
mmc --cc gcc --cflags "-I/foo/bar" example.m
Just as ghc currently provides the OPTIONS pragma {-# OPTIONS -fallow-overlapping-instances #-} to control Haskell compiler flags within the source, I can imagine a {-# FOREIGN OPTIONS Wibble -fallow-flubble-mangling #-} or similar for foreign compiler calls. Such pragmas may not be required, but they should be possible.
I certainly consider the mechanism that Mercury uses to be elegant enough to be the official Mercury FFI, and I hope our standards are not so much lower than yours ;-)
I agree. In fact, I was mighty impressed to see that you'd already implemented my proposal in Mercury before I even proposed it. :-) Comments are eagerly solicited. Regards, Malcolm
Wed, 6 Dec 2000 16:35:51 +0000, malcolm-ffi@cs.york.ac.uk <malcolm-ffi@cs.york.ac.uk> pisze:
I *really* *really* want to be able to insert small snippets of foreign code into the source of a Haskell module.
You can do it with hsc2hs: foreign import "hs_PyTuple_Check" unsafe tupleCheck :: Object -> Bool #def inline HsBool hs_PyTuple_Check (PyObject *o) \ {return PyTuple_Check (o);} The C wrapper was needed here because PyTuple_Check is a macro. Inside the body there is no need of \ at the end of every line. Until the matching } everything belongs to the #def.
I know that it is always possible to write a small .c file and compile and link it in separately, but with the new FFI that now seems to necessitate writing an additional .h file as well.
hsc2hs writes the .h and .c files for you. It knows how to extract a declaration or prototype from a definition of a function (inline or not), variable, struct and typedef. You have to compile and link the .c file yourself.
For instance, in the new FFI I often want to do something like
foreign code { #include <dirent.h> }
Includes are handled separately. You write #include <dirent.h> and it gets included where needed: into the temporary C program which outputs the Haskell source, in the compiled Haskell module if compiling via C, and into the .h file if it was generated because a #def was present.
foreign code C { char *readdirent (DIR*d) { struct dirent *e; e = readdir(d); return e->d_name; } }
With hsc2hs you don't have to write such wrappers for all fields. (#peek struct dirent, d_name) is an expression which can be used as having the type Ptr a -> IO (Ptr CChar) You have to ensure that its usage allows to infer the right type for the result. It is overloaded on Storable.
As people have pointed out, we need to avoid being totally C-centric, which is why there needs to be a language identifier attached.
hsc2hs is of course C centric, but for another language simply a different preprocessor would be used. Anyone wants to design hsc++2hs? :-) But currently you would have to write extern "C" wrappers for all native C++ functions, methods, constructors, destructors, conversions, casts through virtual inheritance... -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
On 06-Dec-2000, Marcin 'Qrczak' Kowalczyk <qrczak@knm.org.pl> wrote:
As people have pointed out, we need to avoid being totally C-centric, which is why there needs to be a language identifier attached.
hsc2hs is of course C centric, but for another language simply a different preprocessor would be used.
Anyone wants to design hsc++2hs? :-) But currently you would have to write extern "C" wrappers for all native C++ functions, methods, constructors, destructors, conversions, casts through virtual inheritance...
Perhaps should talk about that the approach I intend to take to allow Mercury's FFI to interface to multiple languages. This is intended to avoid having to write extern "C" wrappers by hand. Each backend (e.g. gcc C code, Java, MS IL) will have a "preferred" foreign language. If the compiler actually compiles to this language it could be possible to inline foreign language code snippets (this happens with the C backend, and could happen with IL assembler or Java foreign code too). The compiler knows how to interface Mercury code to this preferred foreign language. For ("preferred" language, foreign language) pair, you define an "extrusion". This defines how to generate the appropriate code to interface with one foreign language in another. For example, for predicate/function `foo' with an implementation written in C++ code, with a C code backend: - put the `foo' foreign code snippet inside a C++ function `foo_cpp' with C linkage (e.g. use extern "C" wrappers around the prototype) - generate a foreign code snipped `foo_c' that calls C++ function `foo_cpp' (so you need to be able to generate a call in C) - use `foo_c' as the implementation of `foo'. After extrusion, all remaining C++ code gets shoved into a .cpp file, which will be compiled and linked. The trick of automatically putting the code into a separate file, generating the correct linkage and calling it should work fine for most compiled languages if C is available as a backend. Note that each C++ fragment gets translated into a new C++ fragment plus a C fragment. Say you wanted to do the same with Python: - generate a foreign_decl("C", "#include <Python.h>") - generate a C fragment that looks something like: PyObject *locals, *globals, *result; char * pythoncode = "... code for foo, using local variables as parameters ..."; locals = ... code for marshalling parameters into locals ... globals = ... any useful globals ... result = PyRun(pythoncode, 0, globals, locals); ... marshall result if necessary ... - use this C fragment as the implementation There are of course different ways of doing this that have different trade-offs, but this approach will work pretty well for any interpreter with a C API. Note that each Python fragment gets translated into a single C fragment -- the python code disappears into a C string. After extrusion the Mercury compiler will never see any python code, just C code. The build environment would define compilers, flags, link options, etc. The exact supported languages and combinations is implementation specific (and installation specific -- if you don't have the compilers, obviously it's not going to work). The compiler needs to know how to interface Mercury to a single "preferred" foreign language (probably one per backend). The compiler (or external tool or plugin) also needs to know how to generate enough of this "preferred" foreign language to call each supported non-preferred language. Finally it needs to know what to do with any "left-over" non-preferred code after extrusion (e.g. put it into a separate file). The neat thing is that you should be able to chain the extrusions. E.g. if we write code to let the Mercury compiler extrude C into Java (so we can use C code in the Java backend), then we can extrude Python to C and hence we can call Python from Java. Getting the compile flags and link options right for all of this could be quite tricky, of course, because build and link tools are rarely orthogonal enough to handle this nicely. We also need to handle which language to prefer if there are multiple workable implementations available. This is a research proposal that is half implemented at the moment (the extrusion is still not quite finished). -- Tyson Dowd # # Surreal humour isn't everyone's cup of fur. trd@cs.mu.oz.au # http://www.cs.mu.oz.au/~trd #
On 06-Dec-2000, malcolm-ffi@cs.york.ac.uk <malcolm-ffi@cs.york.ac.uk> wrote:
Fergus writes, triggered by my suggestion of "foreign value":
[ stuff deleted ]
I propose that foreign code <language-id> { <code> } should simply cause the verbatim copying of the <code> part into a file to be compiled by the appropriate foreign compiler. That's all. Nothing fancy.
Actually I should point out that sometimes the Mercury FFI does do something a little bit fancy at this point. First is that for the Managed C++ backend (an extension of C++ to allow it to generate garbage collected types and bytecode for the Microsoft .NET Frameworks), it will automatically create a class and will insert foreign_decls above the class definition, and foreign_codes inside the class definition. Any predicate/function implementations are created as static methods of the class.
:- pragma foreign_decls(c, " #define _POSIX_SOURCE 1 extern int my_global; ").
Fergus, could you enlighten me some more about the difference between the foreign_decls and foreign_code pragmas? I'm afraid I can't immediately see why you have both.
There is a difference in the way such code is treated with respect to optimization. We haven't formalized the exact definition of the difference just yet -- this is an outstanding bug in our documentation. The implementation is free to split each foreign_code fragment into (for example) a separate file for each fragment, or to cross-module inline such fragments. Mercury can do both of these optimizations. But such code might need to share a common set of declarations. So foreign_decls can be viewed as the "common" declarations for all foreign code in that module. Each output file that uses a foreign_code fragment must also include (possibly by #include, possibly by textual subsitution) the foreign_decls part. The declarations in the foreign_decls must not define any storage. In C they will correspond very closely to a "header file". The definitions in foreign_code must not rely upon definitions in any other foreign_code. They can rely on declarations in the foreign_decls, however.
the Mercury compiler extracts the foreign_decl and foreign_code stuff, wrapping code in the second form of foreign_code up inside appropriate function definitions, and puts them all in a separate file that it then invokes the foreign language compiler on.
And I would expect a Haskell compiler to do something similar. For C, Java, and the like, this would be fairly easy for the compiler writer to provide. But in the general case, I'd guess that there should be some sort of configurable "plugin" interface, by which an ordinary user could tell the the Haskell compiler about foreign language name: Wibble compiler name: wblc default options: -fno-flubble-mangling extra linking options: -lwibblelib and such like, so that they could use the FFI for whatever language they pleased.
This would (possibly) be enough to let them use whatever *compiler* they want, but not necessarily whatever *language*. Different languages have different calling conventions and marshalling requirements. I would expect you also need a programmatic interface to allow plugins to be written to handle these (sort of like writing a little code generator for each language).
I certainly consider the mechanism that Mercury uses to be elegant enough to be the official Mercury FFI, and I hope our standards are not so much lower than yours ;-)
I agree. In fact, I was mighty impressed to see that you'd already implemented my proposal in Mercury before I even proposed it. :-)
This is a pretty recent addition to Mercury. We've been thinking about it for a while. The need has come up because as we target new backends we need to re-target the Mercury library, which involves re-writing a lot of C code into {Java, Managed C++, C#, whatever}. Also we found that we were doing almost all of this work anyway in the MCORBA tool, because it interfaced with C++. MCORBA was generating C code to call C++, and C++ code to talk to the CORBA ORB. -- Tyson Dowd # # Surreal humour isn't everyone's cup of fur. trd@cs.mu.oz.au # http://www.cs.mu.oz.au/~trd #
[This comment is a bit late, I know...] malcolm-ffi@cs.york.ac.uk wrote,
Fergus writes, triggered by my suggestion of "foreign value":
Mercury supports this using `foreign_decls' and `foreign_code' pragmas.
I refrained from proposing this in an earlier mail, but I've been thinking about it for a long time. I *really* *really* want to be able to insert small snippets of foreign code into the source of a Haskell module.
GreenCard allows you to do this, via the %- or %C directives. It is one of the features I miss most about the new FFI. I know that it is always possible to write a small .c file and compile and link it in separately, but with the new FFI that now seems to necessitate writing an additional .h file as well. And all for a handful of lines of foreign code that could easily be kept in the Haskell source, with the ease-of-comprehension benefits entailed by keeping everything together that belongs together. [..]
I believe that functionality like this should be implemented by an extra tool rather than the Haskell compiler. GreenCard does it, hsc2hs has something similar, and I am considering a similar facility for c2hs. I completely agree with your motivation of keeping small foreign code snippets inline to make the interface easier to comprehend and maintain. This is, however, a good indicator that we should use an extra tool. IMHO, the FFI language extension should me as small as possible. Convenience should be provided by extra tools. Unless, the feature is impossible (or very hard) to implement as an extra tool, but I am not convinced that this is the case here. Cheers, Manuel
On 02-Jan-2001, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> wrote:
[This comment is a bit late, I know...]
malcolm-ffi@cs.york.ac.uk wrote,
Fergus writes, triggered by my suggestion of "foreign value":
Mercury supports this using `foreign_decls' and `foreign_code' pragmas.
I refrained from proposing this in an earlier mail, but I've been thinking about it for a long time. I *really* *really* want to be able to insert small snippets of foreign code into the source of a Haskell module.
GreenCard allows you to do this, via the %- or %C directives. It is one of the features I miss most about the new FFI. I know that it is always possible to write a small .c file and compile and link it in separately, but with the new FFI that now seems to necessitate writing an additional .h file as well. And all for a handful of lines of foreign code that could easily be kept in the Haskell source, with the ease-of-comprehension benefits entailed by keeping everything together that belongs together. [..]
I believe that functionality like this should be implemented by an extra tool rather than the Haskell compiler.
Doing it in a separate tool will lose efficiency in some important cases. If the compiler is compiling via C, then it can insert inline C code directly in the generated code, and thus get inlining. But I think a separate tool would have to put the C code in a separate C file, which would prevent inlining. -- Fergus Henderson <fjh@cs.mu.oz.au> | "I have always known that the pursuit | of excellence is a lethal habit" WWW: <http://www.cs.mu.oz.au/~fjh> | -- the last words of T. S. Garp.
Tue, 2 Jan 2001 22:28:02 +1100, Fergus Henderson <fjh@cs.mu.oz.au> pisze:
Doing it in a separate tool will lose efficiency in some important cases. If the compiler is compiling via C, then it can insert inline C code directly in the generated code, and thus get inlining. But I think a separate tool would have to put the C code in a separate C file, which would prevent inlining.
hsc2hs creates a .c file and a .h file for snippets of C code. The .h file is included into the .hc file generated by ghc. If you mark functions as inline, they will be really inlined. BTW. gcc understoods the word "inline" differently than ISO/ANSI C99 defines it. The following files are produced from a simple source: #include <errno.h> #def inline HsInt get_errno (void) {return errno;} It works with gcc and should work with a hypothetical C compiler which understands inline correctly. -------- .h file -------- #ifndef TEST_HS_H #define TEST_HS_H #if __GLASGOW_HASKELL__ && __GLASGOW_HASKELL__ < 409 #include <Rts.h> #endif #include <HsFFI.h> #include <errno.h> #ifdef __GNUC__ extern #endif inline HsInt get_errno (void) {return errno;} #endif -------- .c file -------- #include "Test.hs.h" #ifndef __GNUC__ extern #endif inline HsInt get_errno (void) #ifndef __GNUC__ ; #else {return errno;} #endif -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
qrczak@knm.org.pl (Marcin 'Qrczak' Kowalczyk) wrote,
Tue, 2 Jan 2001 22:28:02 +1100, Fergus Henderson <fjh@cs.mu.oz.au> pisze:
Doing it in a separate tool will lose efficiency in some important cases. If the compiler is compiling via C, then it can insert inline C code directly in the generated code, and thus get inlining. But I think a separate tool would have to put the C code in a separate C file, which would prevent inlining.
hsc2hs creates a .c file and a .h file for snippets of C code. The .h file is included into the .hc file generated by ghc. If you mark functions as inline, they will be really inlined.
This sounds like a good solution to the problem. Cheers, Manuel
On 02-Jan-2001, Marcin 'Qrczak' Kowalczyk <qrczak@knm.org.pl> wrote:
Tue, 2 Jan 2001 22:28:02 +1100, Fergus Henderson <fjh@cs.mu.oz.au> pisze:
Doing it in a separate tool will lose efficiency in some important cases. If the compiler is compiling via C, then it can insert inline C code directly in the generated code, and thus get inlining. But I think a separate tool would have to put the C code in a separate C file, which would prevent inlining.
hsc2hs creates a .c file and a .h file for snippets of C code. The .h file is included into the .hc file generated by ghc.
OK, so this is a hybrid approach -- most of the work is done by an external tool, but ghc still needs to know to #include the right .h file in the .hc file. So you don't need `foreign_code', but you still need `foreign_decls' or equivalent. Personally I'd rather have a single well-designed convenient foreign language interface as a standard part of the language, rather than having a minimalistic foreign language interface in the language standard and having convenience provided by a separate tool (or by several competing separate tools). But I understand why you might differ on that point. -- Fergus Henderson <fjh@cs.mu.oz.au> | "I have always known that the pursuit | of excellence is a lethal habit" WWW: <http://www.cs.mu.oz.au/~fjh> | -- the last words of T. S. Garp.
I'm still with Fergus on this one:
OK, so this is a hybrid approach -- most of the work is done by an external tool, but ghc still needs to know to #include the right .h file in the .hc file. So you don't need `foreign_code', but you still need `foreign_decls' or equivalent.
Exactly. Ghc has "foreign decl" by stealth. Hugs, nhc98, and hbc have no equivalent to #include an arbitrary .h file into generated-via-C code. I think we need a standard common language-based mechanism.
Personally I'd rather have a single well-designed convenient foreign language interface as a standard part of the language, rather than having a minimalistic foreign language interface in the language standard and having convenience provided by a separate tool (or by several competing separate tools). But I understand why you might differ on that point.
Indeed, this was one of the major initial motivations to develop a common FFI standard. People had plenty of FFI tools already, but the more tools we have, the more we need a standard language base for the tools to work with. Although I am happy to use tools for generating FFI code, I do *not* want to be *required* to use them to achieve simple functionality. I am about to add something like {-# CCODE #include <stdio.h> #-} to nhc98 as a stop-gap. Yes, this inevitably differs from whatever mechanism ghc currently uses. So yes, tools like hsc2hs will have to know which compiler they are generating for. Unless anyone wants to have a rethink now? :-) Regards, Malcolm
On Wed, 3 Jan 2001 malcolm-ffi@cs.york.ac.uk wrote:
I am about to add something like {-# CCODE #include <stdio.h> #-} to nhc98 as a stop-gap. Yes, this inevitably differs from whatever mechanism ghc currently uses. So yes, tools like hsc2hs will have to know which compiler they are generating for. Unless anyone wants to have a rethink now? :-)
ghc has -#include <stdio.h> command-line option (if the header name is not in "" or <>, "" are assumed). Any command-line options can be put in the source thus: {-# OPTIONS -#include <stdio.h> #-} (These options currently have to be at the very top of the file and AFAIK recently ghc was changed to accept more than one OPTIONS line.) If nhc adopts that way, it must not generate prototypes itself of course. A signature of a C function derived from Haskell is precise enough to call the function in the right way via a native code generator, but not to generate prototypes with exact constness etc. hsc2hs inserts OPTIONS pragma when generating for ghc. QForeign comes with wrappers for some libraries and a script to output hcflags, ldflags, ldlibs. The script outputs appropriate -#include options for requested libraries. Installation also runs ghc --add-package (on ghc >= 4.09) to specify C includes among other things, and then e.g. -package qforeign-curses option causes inclusion of curses.h, which makes the former script unnecessary. Having correct includes is especially important for ghc, because calls to C functions may be inlined across Haskell modules. A user module doing "import Curses" and being compiled with optiimzation and without appropriate -#include options may cause the C compiler to call function without prototypes, which is not legal in C99 and does not work for argument types like char. -- Marcin 'Qrczak' Kowalczyk
Wed, 3 Jan 2001 21:44:03 +1100, Fergus Henderson <fjh@cs.mu.oz.au> pisze:
Personally I'd rather have a single well-designed convenient foreign language interface as a standard part of the language, rather than having a minimalistic foreign language interface in the language standard and having convenience provided by a separate tool (or by several competing separate tools). But I understand why you might differ on that point.
I would prefer that too if it was possible, but it does not look well. Interfacing to C is complex enough, it's yet worse for other languages. A problem: HsFFI includes some C headers (stdint.h, which includes among others features.h on glibc). There are headers like stdio.h (if I remember correctly) which provide certain prototypes or macros only when certain macros are defined, like _XOPEN_SOURCE and _GNU_SOURCE. If I #define these constants after inclusion HsFFI.h, it's too late: features.h already determined what should be visible and it will not be processed again. But hsc2hs does not distinguish between #defines which should go before HsFFI.h and those which should go after. So I had to define them via {-# OPTIONS -optc-D_XOPEN_SOURCE=500 #-} instead of #define, or rather hsc2hs's #option (because it generates its own {-# OPTIONS #-} and only one such pragma used to be accepted by ghc), to be visible early enough. But hsc2hs needs them defined during its job of preprocessing (for constants), so #define is needed too, protected by #undef because symbols will be already defined when #defines are processed in the generated C header (they are put in multiple places): #option -optc-D_GNU_SOURCE -optc-D_XOPEN_SOURCE=500 #undef _GNU_SOURCE #define _GNU_SOURCE 1 #undef _XOPEN_SOURCE #define _XOPEN_SOURCE 500 Ugly, isn't it? Better ugly than impossible, assuming that the compiler provided only a single elegant way of handling #defines. Would a builtin support predict all possible uses of C features? -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
On 03-Jan-2001, Marcin 'Qrczak' Kowalczyk <qrczak@knm.org.pl> wrote:
Wed, 3 Jan 2001 21:44:03 +1100, Fergus Henderson <fjh@cs.mu.oz.au> pisze:
Personally I'd rather have a single well-designed convenient foreign language interface as a standard part of the language, rather than having a minimalistic foreign language interface in the language standard and having convenience provided by a separate tool (or by several competing separate tools). But I understand why you might differ on that point.
I would prefer that too if it was possible, but it does not look well. Interfacing to C is complex enough, it's yet worse for other languages.
A problem: HsFFI includes some C headers (stdint.h, which includes among others features.h on glibc). There are headers like stdio.h (if I remember correctly) which provide certain prototypes or macros only when certain macros are defined, like _XOPEN_SOURCE and _GNU_SOURCE. ... #option -optc-D_GNU_SOURCE -optc-D_XOPEN_SOURCE=500 #undef _GNU_SOURCE #define _GNU_SOURCE 1 #undef _XOPEN_SOURCE #define _XOPEN_SOURCE 500
Ugly, isn't it? ... Would a builtin support predict all possible uses of C features?
Well, the Mercury language specification doesn't say anything about compiler options, since they tend to vary between implementations. But with the Mercury build environment (a de facto standard, I suppose ;-), you can solve that problem by putting CFLAGS = -D_GNU_SOURCE -D_XOPEN_SOURCE=500 in your Mmakefile, or CFLAGS-foo = -D_GNU_SOURCE -D_XOPEN_SOURCE=500 if you only want it for module `foo'. -- Fergus Henderson <fjh@cs.mu.oz.au> | "I have always known that the pursuit | of excellence is a lethal habit" WWW: <http://www.cs.mu.oz.au/~fjh> | -- the last words of T. S. Garp.
Fergus Henderson <fjh@cs.mu.oz.au> wrote,
Personally I'd rather have a single well-designed convenient foreign language interface as a standard part of the language, rather than having a minimalistic foreign language interface in the language standard and having convenience provided by a separate tool (or by several competing separate tools). But I understand why you might differ on that point.
I doubt that you want to support the functionality of H/Direct or C->HS in the standard part of the language, which means that without a tool you have to settle for less anyway. Or in other words, you will use a tool anyway for getting a really convenient interface. If so, why not move as much of the work to the tool as possible? Cheers, Manuel
Fergus Henderson <fjh@cs.mu.oz.au> wrote,
On 02-Jan-2001, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> wrote:
[This comment is a bit late, I know...]
malcolm-ffi@cs.york.ac.uk wrote,
Fergus writes, triggered by my suggestion of "foreign value":
Mercury supports this using `foreign_decls' and `foreign_code' pragmas.
I refrained from proposing this in an earlier mail, but I've been thinking about it for a long time. I *really* *really* want to be able to insert small snippets of foreign code into the source of a Haskell module.
GreenCard allows you to do this, via the %- or %C directives. It is one of the features I miss most about the new FFI. I know that it is always possible to write a small .c file and compile and link it in separately, but with the new FFI that now seems to necessitate writing an additional .h file as well. And all for a handful of lines of foreign code that could easily be kept in the Haskell source, with the ease-of-comprehension benefits entailed by keeping everything together that belongs together. [..]
I believe that functionality like this should be implemented by an extra tool rather than the Haskell compiler.
Doing it in a separate tool will lose efficiency in some important cases. If the compiler is compiling via C, then it can insert inline C code directly in the generated code, and thus get inlining. But I think a separate tool would have to put the C code in a separate C file, which would prevent inlining.
This has some problematic consequences: * Code which replies on this inlining would require any high-performance Haskell compiler to compile via C. * As I understand, these pragmas are supposed to support not only C. How about C++ code? Will there be a C++ backend for Mercury just to make this code efficient? Cheers, Manuel
On 03-Jan-2001, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> wrote:
Fergus Henderson <fjh@cs.mu.oz.au> wrote,
Doing it in a separate tool will lose efficiency in some important cases. If the compiler is compiling via C, then it can insert inline C code directly in the generated code, and thus get inlining. But I think a separate tool would have to put the C code in a separate C file, which would prevent inlining.
This has some problematic consequences:
* Code which replies on this inlining would require any high-performance Haskell compiler to compile via C.
No code should rely upon this inlining for correctness. AFAIK none of it does. One of our backends that compiled via C had this sort of inlining turned off for quite some time.
* As I understand, these pragmas are supposed to support not only C. How about C++ code? Will there be a C++ backend for Mercury just to make this code efficient?
No. There's only (at most) one language per backend that can be inlined in this way. Otherwise you have to do your best. You can inline the C wrapper code that is used to call into C++, but that's about it. The inlining is not important for semantics, but it might be important for benchmarks ;-) -- Tyson Dowd # # Surreal humour isn't everyone's cup of fur. trd@cs.mu.oz.au # http://www.cs.mu.oz.au/~trd #
On 03-Jan-2001, Tyson Richard DOWD <trd@cs.mu.OZ.AU> wrote:
On 03-Jan-2001, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> wrote:
* As I understand, these pragmas are supposed to support not only C. How about C++ code? Will there be a C++ backend for Mercury just to make this code efficient?
No. There's only (at most) one language per backend that can be inlined in this way. Otherwise you have to do your best. You can inline the C wrapper code that is used to call into C++, but that's about it.
Actually perhaps you can if the `extern inline' trick works across C and C++. But in general you can't. For example you'll probably never be able to inline Java from a C backend. -- Tyson Dowd # # Surreal humour isn't everyone's cup of fur. trd@cs.mu.oz.au # http://www.cs.mu.oz.au/~trd #
On 03-Jan-2001, Tyson Richard DOWD <trd@cs.mu.OZ.AU> wrote:
On 03-Jan-2001, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> wrote:
* As I understand, these pragmas are supposed to support not only C. How about C++ code? Will there be a C++ backend for Mercury just to make this code efficient?
No. There's only (at most) one language per backend that can be inlined in this way.
Actually that's not necessarily true. For example, the current Mercury compiler compiles via C and then via asm. We could probably support both inline C and inline asm. Currently we don't bother to support inline asm, since (a) we've never needed it and (b) you can get the same effect by writing inline asm inside your inline C (presuming the C compiler supports inline asm), but I wouldn't be too surprised to see it in the future. I also wouldn't be surprised to see a C++ backend for Mercury. Given that we already support a C back-end, a C++ back-end would not be difficult to implement.
Otherwise you have to do your best. You can inline the C wrapper code that is used to call into C++, but that's about it.
The way to make the inlining trick work across multiple languages is to provide either (a) a single definition that works in multiple languages, and then inline it into all those languages or (b) provide different definitions for different languages, and then use the appropriate version depending on which target language you are compiling to. For those who are not fans of polyglot-style obfuscated programs, (a) generally only works in the case of languages which are syntactically quite similar, such as C and C++, where you can easily write code that is in the intersection of both languages. -- Fergus Henderson <fjh@cs.mu.oz.au> | "I have always known that the pursuit | of excellence is a lethal habit" WWW: <http://www.cs.mu.oz.au/~fjh> | -- the last words of T. S. Garp.
Tyson Dowd <trd@cs.mu.OZ.AU> wrote,
On 03-Jan-2001, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> wrote:
Fergus Henderson <fjh@cs.mu.oz.au> wrote,
Doing it in a separate tool will lose efficiency in some important cases. If the compiler is compiling via C, then it can insert inline C code directly in the generated code, and thus get inlining. But I think a separate tool would have to put the C code in a separate C file, which would prevent inlining.
This has some problematic consequences:
* Code which replies on this inlining would require any high-performance Haskell compiler to compile via C.
No code should rely upon this inlining for correctness.
That wasn't my point - I guess, I should have been more clear about this. Having a language feature whose efficient implementation requires a compiler to compile, say, via C is a problem if somebody decides tomorrow to implement a Haskell compiler directly generating assembler code. The decision is easier for you, because there is only one Mercury compiler. Cheers, Manuel
On 04-Jan-2001, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> wrote:
Having a language feature whose efficient implementation requires a compiler to compile, say, via C is a problem if somebody decides tomorrow to implement a Haskell compiler directly generating assembler code.
No, it's not a problem, it's a feature. There's two choices: - completely disallow inter-language inlining (inefficient all the time) - allow inter-language inlining in some cases (efficient some of the time, but inefficient at other times) The ideal, i.e. - allow inter-language inlining in ALL cases (efficient all of the time) is not possible. But that doesn't mean that we should therefore choose to always completely disallow inter-language inlining. Rather, we should make it possible for implementations to offer that feature. Not all implementations will take advantage of this possibility, because there are trade-offs involved, and some implementations will want to offer different trade-offs. But that's OK. By allowing the possibility, we give programmers more choice. Different choices will be better for different applications, so giving the programmer the choice is a Good Thing.
The decision is easier for you, because there is only one Mercury compiler.
There's only one front-end, but there's lots of back-ends. Only two of them (less than half) compile via C. But I want inline C to be fast in the one that compiles via C, and I want inline Java to be fast in the Java back-end. -- Fergus Henderson <fjh@cs.mu.oz.au> | "I have always known that the pursuit | of excellence is a lethal habit" WWW: <http://www.cs.mu.oz.au/~fjh> | -- the last words of T. S. Garp.
I refrained from proposing this in an earlier mail, but I've been thinking about it for a long time. I *really* *really* want to be able to insert small snippets of foreign code into the source of a Haskell module.
I *really* *really* don't want this. The current Hugs implementation of ffi could readily accomodate it because it generates a C file which you compile in the normal way but it would forever rule out an improved implementation (for use on especially well supported architectures) which doesn't need a compiler at all. One could argue that it'd be ok to support it as an optional extension on just some platforms and people would just have to be careful to use the portable subset when appropriate. Anyone who has tried to compile Linux code (intended for gcc) on some other compiler knows that people don't restrict themselves to the portable subset and, after a while, forget whether they're using extensions at all. In other words, I'd like to keep all ffi implementations identical: no non-portable extensions. [Note that the same portability argument applies to supporting unboxed pointers in the ffi:not portable and not necessary.]
Doing it in a separate tool will lose efficiency in some important cases. If the compiler is compiling via C, then it can insert inline C code directly in the generated code, and thus get inlining. But I think a separate tool would have to put the C code in a separate C file, which would prevent inlining.
My group at University of Utah will be releasing "Knit" in a week or two. Knit is mostly an adaptation/implementation of Matthew Flatt's component programming model for C (see his PLDI'98 paper). But Knit can also merge multiple C files into one file, flag functions for inlining (using a simple cost model to avoid exponential code size increase), and sort function definitions so that definitions come before uses (required by gcc's inliner). If you're really that obsessed with performance, you could use Knit :-) [Knit is, of course, written in Haskell.] Or, if you want a lower-tech solution, you could define all those little functions as inline functions in a .h file and use whatever flags or directives nhc/ffi provides to say "please #include this .h file". With the latter approach, we might want to tweak GreenCard and friends to optionally flag all the internal C functions it generates as "static inline" to enhance inlining when used this way. The "static" annotation would have to be optional (i.e., controlled by an ifdef) because we'd also want to be able to compile the file by itself. Alastair
On 03-Jan-2001, Alastair Reid <reid@cs.utah.edu> wrote:
I refrained from proposing this in an earlier mail, but I've been thinking about it for a long time. I *really* *really* want to be able to insert small snippets of foreign code into the source of a Haskell module.
I *really* *really* don't want this.
The current Hugs implementation of ffi could readily accomodate it because it generates a C file which you compile in the normal way but it would forever rule out an improved implementation (for use on especially well supported architectures) which doesn't need a compiler at all.
What would be the improvement? The fact that it doesn't need a C compiler? If you don't have a C parser, you can't interface to C macros or enumeration constants, and you can't do type-safe cross-language interfacing. I don't see how that can be considered an improvement. I also don't see how duplicating the C parser in a separate tool is an improvement either, especially considering that the separate tool will need to be updated to keep track of changes in C as it evolves. For cases where someone wants to write inline code in C or <insert language here>, they will need a compiler for that language. This is true regardless of whether they write the foreign code inline in the Haskell file or in a separate file. And it's easy enough to implement the other FFI constructs that don't involve inline foreign code in such a way that if you don't use inline foreign code, you don't need a foreign language compiler. -- Fergus Henderson <fjh@cs.mu.oz.au> | "I have always known that the pursuit | of excellence is a lethal habit" WWW: <http://www.cs.mu.oz.au/~fjh> | -- the last words of T. S. Garp.
Fergus Henderson <fjh@cs.mu.oz.au> wrote,
On 03-Jan-2001, Alastair Reid <reid@cs.utah.edu> wrote:
I refrained from proposing this in an earlier mail, but I've been thinking about it for a long time. I *really* *really* want to be able to insert small snippets of foreign code into the source of a Haskell module.
I *really* *really* don't want this.
The current Hugs implementation of ffi could readily accomodate it because it generates a C file which you compile in the normal way but it would forever rule out an improved implementation (for use on especially well supported architectures) which doesn't need a compiler at all.
What would be the improvement? The fact that it doesn't need a C compiler?
If you don't have a C parser, you can't interface to C macros or enumeration constants, and you can't do type-safe cross-language interfacing. I don't see how that can be considered an improvement.
I also don't see how duplicating the C parser in a separate tool is an improvement either, especially considering that the separate tool will need to be updated to keep track of changes in C as it evolves.
You should consider that the situation fgor Mercury is slightly different from that for Haskell, as there is only one Mercury compiler. This makes merging tool support into the compiler infrastructure more attractive. If there is more than one compiler/interpreter for a language, duplicating a C parser in an external tool can prevent duplicating work in the various language implementations - which I prefer anytime, because Haskell implementations are very complex already anyway. So, IMHO, if a feature can be reasonably implemented in an extra tool, I'd rather do that than add it to the language core. I think, Marcin's inlining scheme demostrates that we can get the inlining even with an extra tool, so I don't see a compelling reason for making this part of the language. Cheers, Manuel
Anyone who has tried to compile Linux code (intended for gcc) on some other compiler knows that people don't restrict themselves to the portable subset and, after a while, forget whether they're using extensions at all.
Or indeed, anyone who has tried to compile Haskell code intended for GHC. I am constantly surprised by what people think is "standard" Haskell. Standards are GOOD. Extensions are BAD.
... use whatever flags or directives nhc/ffi provides to say "please #include this .h file".
Ah yes, the point is, neither nhc98 nor the common FFI has any mechanism for saying "please #include this .h file". I want one. And I want a *standard* one. foreign decl CPP { #define _GNU_SOURCE #define _XOPEN_SOURCE=500 #include <stdio.h> #include "HsFFI.h" }
The current Hugs implementation of ffi could readily accomodate it because it generates a C file which you compile in the normal way but it would forever rule out an improved implementation (for use on especially well supported architectures) which doesn't need a compiler at all.
So you shift the burden of calling the C compiler out of the Haskell system, and onto the user to do it manually? Really, I'd rather the Haskell system tried its best to do as much as possible automatically. Regards, Malcolm
On 28-Nov-2000, Simon Marlow <simonmar@microsoft.com> wrote:
qrczak@knm.org.pl (Marcin 'Qrczak' Kowalczyk) wrote,
-- Wrappers that convert the result to a possible exception. -- *_EINTR variants repeat the action if there was an error -- and errno == EINTR. throwIf :: (a -> Bool) -> String -> IO a -> IO a throwIf_EINTR :: (a -> Bool) -> String -> IO a -> IO a
throwIfMinus1 :: Num a => String -> IO a -> IO a throwIfMinus1_EINTR :: Num a => String -> IO a -> IO a throwIfNull :: String -> IO (Ptr a) -> IO (Ptr a) throwIfNull_EINTR :: String -> IO (Ptr a) -> IO (Ptr a)
Again, this belongs into the higher-level marshalling library. Did you have a look at the corresponding routines in C2HS?
Your ifRaise family is fairly similar (except that it doesn't deal explicitly with C's errno and EINTR). I don't particularly mind what these functions are called or where they live, as long as we have them!
Admittedly the functions above are C-specific and should probably go into a C-specific library on top of Foreign. CForeign anyone?
These are not C-specific, they are POSIX-specific. As evidence that they are not C-specific, I offer the fact that the Mercury "extras" distribution includes a POSIX interface library which defines `errno' and `eINTR'. No doubt Perl's POSIX interface also defines equivalents. -- Fergus Henderson <fjh@cs.mu.oz.au> | "I have always known that the pursuit | of excellence is a lethal habit" WWW: <http://www.cs.mu.oz.au/~fjh> | -- the last words of T. S. Garp.
On Tue, 28 Nov 2000, Simon Marlow wrote:
arguably, withObject and newObject should be named with and new ("Object" is awfully vague and seems redundant anyway).
Indeed, but "with" is a keyword in -fglasgow-exts and Hugs :-( I'm not happy with the fact that some with* functions look inside the named thing (withForeignObj, also withHandle in ghc's internals) and some do the reverse: build a temporary object of the said type. I tried allocaPoke* and mallocPoke* but they are quite ugly (allocaPokeCString is the most basic way to pass strings to C functions). Good ideas are welcome.
Admittedly the functions above are C-specific and should probably go into a C-specific library on top of Foreign. CForeign anyone?
Unfortunately it's hard to completely separate C-specific things. Is malloc C-specific? Theoretically yes, but I doubt that we want a generic allocation function which does not necessarily match C's malloc in the generic part, and additionally C's malloc in the C-specific part. I'm afraid that given the reality we live in, the basic part must refer to C sometimes. E.g. we should guarantee that function pointers obtained from foreign export dynamic using the default calling convention are compatible with standard C function pointers, and that foreign import uses C's name mangling by default. We may separate some truly C-specific things to make the generic module smaller (e.g. errno translation), but it's not going to be a pure separation of only generic things vs. things having something common with C. It will bite when trying to interface to C++'s object allocation. But there are bigger problems there anyway - allocation is normally coupled with initialization, and destructors should be run by default before freeing - so let's stick with C's malloc in the generic part, and we will need other mechanisms for more complex languages.
Meanwhile we should have some non-C-specific error-handling utilities in Foreign.
IMHO there is little or nothing to place there, assuming that the module Exception is available. The hard-to-do-portably thing is to convert all errno values to exceptions, which is C-specific. The only possible generic things are really primitive and would be not used directly anyway: each common error code convention creates the need of wrappers specific to that convention. It would be good to provide specific wrappers in specialized modules, e.g. handling some common Windows error codes. -- Marcin 'Qrczak' Kowalczyk
participants (8)
-
Alastair Reid -
Fergus Henderson -
malcolm-ffi＠cs.york.ac.uk -
Manuel M. T. Chakravarty -
Marcin 'Qrczak' Kowalczyk -
qrczak＠knm.org.pl -
Simon Marlow -
Tyson Dowd