Ok FFI folks, I think we're all agreed now on the desirability of tweaking the FFI to permit (a) arbitrary new foreign languages to be added; (b) the "foreignname" specifier to be (potentially) richer, depending on the particular foreign language in question; (c) abstraction of common header/linklib information to a single new sort of decl. For backwards compatibility, I suppose on (b) we are going to have to continue to allow double-string specifiers of the form foreign import "linklib" "cname" hname :: type for some time, although that form would be deprecated from now on? We know mostly what the richer form of the "foreignname" specifier will be for Java, and in any case we can leave that design to SPJ+EM. However, we have also heard one or two half-hearted suggestions to enrich the "foreignname" string for C as well. For instance, Manuel suggested "Gtk: gtk_new_window". Is anyone convinced that this is a good route for C, or should we settle instead on getting (c) right, leaving the "foreignname" untouched for now? As far as (a) is concerned, we have three decisions to make: (1) Do we use quotes around the language name or accept a simple varid? E.g. "java" vs java. As Marcin points out, the names of the languages C and C++ are not varids, so would need to be enclosed in quotes. On the other hand, see decision point (2) below. (2) The actual names for languages or calling conventions. Manuel suggests a change from naming the calling convention to naming the foreign language. This is somewhat attractive. However, I think many languages have a simple C-callable interface and we don't need to know anything more about the actual language. If this is true, I think we should continue to name the calling convention rather than the language. Hence, the names ccall and stdcall might be retained, java can be added. If anyone actually comes up with a C++ binding, a name like cplusplus (for instance) seems good enough to me. This also has the advantage of backwards compatibility, which incidentally would force decision point (1) above towards no-quotes. Incidentally, the intention would _not_ be that all compilers must support all foreign languages. As at present, only the ccall convention is mandatory. Any other named calling convention should be parsed, but if the compiler doesn't support it, a warning (at least) should be issued. In that situation, it is unclear whether the compiler can actually generate sensible code. (At the moment, I think the FFI standard says if your platform doesn't support stdcall, code should default to ccall? Obviously that doesn't scale well to java for instance!) (3) Where do we allow the language specifier? In my opinion, definitely in foreign import and export decls. This is at minimum for backwards compatibility, but looking forward to the day when someone writes FFI code for more than one language in the same module (eek!), it will be needed there. For the same reason, I would argue that it belongs in the new kind of decl required for (c). Now how do we tackle the biggest change, issue (c): abstraction of common header/linklib information to a single new sort of decl? Disregarding syntax for the moment, Marcin has suggested a concrete design that conflicts in some ways with backwards compatibility: * the primary specification is of the language not the calling convention * the calling convention is defined as only one attribute of a language * the calling convention is mandatorily removed from its current position in all functions His proposal also adds many of the features that we want: * names of include files * names of link libraries * decl appears once and scopes over all foreign functions in that module * it is not language specific: the attribute-based style allows new languages to have attributes we haven't even thought of yet. and some features that might be nice additions: * the global decl can be overridden for a single function * using an attribute style directly in the existing foreign import/export decls is potentially cleaner, as it can tidy up current blots like 'unsafe', and admits the possibility of needing further ad-hoc specifications in the future. (Didn't someone have an example that involved blocking/non-blocking thread-safety?) Since I believe backward compatibility is very important, would anyone like to either modify Marcin's proposal, or make a new concrete proposal that is perhaps slightly more conservative but still achieves our goals? I suppose backwards compatibility might not be entirely possible. If it isn't, then we have a choice between making the changes as small as possible (to make porting old code easier), or making them as radical as possible (to ensure that the design is really clean and extensible). Regards, Malcolm
Tue, 20 Feb 2001 18:54:05 +0000, malcolm-ffi@cs.york.ac.uk <malcolm-ffi@cs.york.ac.uk> pisze:
Since I believe backward compatibility is very important, would anyone like to either modify Marcin's proposal, or make a new concrete proposal that is perhaps slightly more conservative but still achieves our goals?
One point: the syntax should be such that adding future languages, calling conventions, attributes etc. does not break old code which uses these names as e.g. foreign function names. Since sets of languages, attributes etc. are open, it's advantageous when foreign declarations can be parsed without knowledge about these sets. It's for better error messages ("I don't support calling convention foo" vs. "parse error"), for the ability to ignore unknown attributes (should all attributes be ignorable, with a warning? or should they be explicitly marked as ignorable or not? I think we may specify that all are ignorable, for simplicity), and for parsing preprocessors which output foreign declarations unmodified. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
malcolm-ffi@cs.york.ac.uk wrote,
I think we're all agreed now on the desirability of tweaking the FFI to permit
(a) arbitrary new foreign languages to be added;
(b) the "foreignname" specifier to be (potentially) richer, depending on the particular foreign language in question;
(c) abstraction of common header/linklib information to a single new sort of decl.
For backwards compatibility, I suppose on (b) we are going to have to continue to allow double-string specifiers of the form
foreign import "linklib" "cname" hname :: type
for some time, although that form would be deprecated from now on?
Yes, that's how I see it.
We know mostly what the richer form of the "foreignname" specifier will be for Java, and in any case we can leave that design to SPJ+EM. However, we have also heard one or two half-hearted suggestions to enrich the "foreignname" string for C as well. For instance, Manuel suggested "Gtk: gtk_new_window". Is anyone convinced that this is a good route for C, or should we settle instead on getting (c) right, leaving the "foreignname" untouched for now?
We might be able to combine the two - see below.
As far as (a) is concerned, we have three decisions to make:
(1) Do we use quotes around the language name or accept a simple varid? E.g. "java" vs java. As Marcin points out, the names of the languages C and C++ are not varids, so would need to be enclosed in quotes. On the other hand, see decision point (2) below.
(2) The actual names for languages or calling conventions. Manuel suggests a change from naming the calling convention to naming the foreign language. This is somewhat attractive. However, I think many languages have a simple C-callable interface and we don't need to know anything more about the actual language. If this is true, I think we should continue to name the calling convention rather than the language. Hence, the names ccall and stdcall might be retained, java can be added. If anyone actually comes up with a C++ binding, a name like cplusplus (for instance) seems good enough to me. This also has the advantage of backwards compatibility, which incidentally would force decision point (1) above towards no-quotes.
I think, I retract my proposal to change ccall to c (you convinced me). Let's just stick with the simple varids.
Incidentally, the intention would _not_ be that all compilers must support all foreign languages. As at present, only the ccall convention is mandatory. Any other named calling convention should be parsed, but if the compiler doesn't support it, a warning (at least) should be issued. In that situation, it is unclear whether the compiler can actually generate sensible code. (At the moment, I think the FFI standard says if your platform doesn't support stdcall, code should default to ccall? Obviously that doesn't scale well to java for instance!)
I think, when the compiler supports an unsupported calling convention it should abort. stdcall defaulting to ccall is IMHO just a hack.
(3) Where do we allow the language specifier? In my opinion, definitely in foreign import and export decls. This is at minimum for backwards compatibility, but looking forward to the day when someone writes FFI code for more than one language in the same module (eek!), it will be needed there. For the same reason, I would argue that it belongs in the new kind of decl required for (c).
I agree. In fact, I quite liked one of the proposals that you made earlier, namely foreign library C "<ncurses/curses.h>" "-lcurses" foreign library java "blah" "blah" foreign import C "curses_function" cf :: Int -> IO Int foreign import java "java.blah.*/foo" jf :: Int -> IO Int I also like Marcin's proposal, but I guess it is just too far away from what we have and quite same work to implement. So, how about the following. Keep everything string based. A module can have one "foreign defaults" declaration per calling convention, which specifies a string. foreign defaults ccall "<gtk/gtk.h>: " foreign defaults java "java." Now all foreign declarations that want to inherit this string have their "foreignname" string starting with a '*' (or whatever symbol you prefer) foreign import ccall "*gtk_new_window" newWindow :: CInt -> IO (Ptr Window) foreign import java "*blah.foo" jf :: Int -> IO Int Otherwise, they can override the default by using a full specification: foreign import ccall "<glib/glib.h>: g_timer_start" timerStart :: Ptr Timer -> IO () It would be nicer to do without a special symbol indicating the use of the default, but I guess, this will always lead to ambiguous situations. This is, of course, a lot less flexible than what Marcin proposed, but it would be very simple to implement and is more or less backwards compatible. I am not so sure anymore about the specification of link libraries for C. It wouldn't be a problem, because we can always extend the "foreignname", eg, something like "<glib/glib.h>; glib: g_timer_start" However, it usually doesn't help much, to link to GTK+, you need to specify use -L/usr/lib -L/usr/X11R6/lib -lgtk -lgdk -rdynamic -lgmodule -lglib -ldl -lXi -lXext -lX11 -lm which doesn't seem attractive to include in the Haskell source. Moreover, the flags are system dependent. They also can't be cleanly handled with #ifdefs, as the only guarantee that GTK+ gives is that when you execute the command `gtk-config --libs', it will give you all the right options. So, as Marcin pointed out, the only use for a library object spec for ccall is so that interpreters know which handle to pass to dlsym(). I am not too fond of the idea that the interpreter has to try a dlsym() on all library objects that it did dlopen(). Or is this maybe not too bad? Cheers, Manuel
Wed, 21 Feb 2001 16:01:48 +1100, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
So, how about the following.
I must admit that I like my own proposal better :-)
So, as Marcin pointed out, the only use for a library object spec for ccall is so that interpreters know which handle to pass to dlsym(). I am not too fond of the idea that the interpreter has to try a dlsym() on all library objects that it did dlopen(). Or is this maybe not too bad?
As I see in ghc/rts/Linker.c, in the ELF case dlopen (NULL, RTLD_LAZY) and dlopen (filename, RTLD_NOW | RTLD_GLOBAL) can be used to let the linker find the library for each symbol. I don't know about Windows DLLs. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
So, as Marcin pointed out, the only use for a library object spec for ccall is so that interpreters know which handle to pass to dlsym().
This may be the only use but I think it's a very important use.
I am not too fond of the idea that the interpreter has to try a dlsym() on all library objects that it did dlopen(). Or is this maybe not too bad?
I think it's just asking for trouble (for reasons given in earlier mail). -- Alastair
"Alastair Reid" <reid@cs.utah.edu> wrote,
So, as Marcin pointed out, the only use for a library object spec for ccall is so that interpreters know which handle to pass to dlsym().
This may be the only use but I think it's a very important use.
I am not too fond of the idea that the interpreter has to try a dlsym() on all library objects that it did dlopen(). Or is this maybe not too bad?
I think it's just asking for trouble (for reasons given in earlier mail).
Marcin just pointed out that you can use dlopen() in a way that it finds symbols without a library name. That should do, I think. Cheers, Manuel
Marcin just pointed out that you can use dlopen() in a way that it finds symbols without a library name. That should do, I think.
Note that finding a marginally easier way to do the lookup doesn't address my concern that this undirected search will cause maintenance and porting problems by worsening an already bad situation (just which library did Linux/solaris/freebsd/hpux/redhat6.0/redhat7.0/... choose to hide a symbol in). In fact, I expect that having dlopen do a search through the libraries that it thinks are relevant instead of Hugs performing a search through the list of libraries that it knows to be relevant probably makes things worse because it increases the gap between the programmer's mental model and what actually happens. Alastair ps dlopen is not completely portable - Hugs also uses shl_{load,findsym} (HPUX only?) and {LoadLibrary, GetProcAddress} (Win32).
Thu, 22 Feb 2001 11:27:20 -0700, Alastair Reid <reid@cs.utah.edu> pisze:
In fact, I expect that having dlopen do a search through the libraries that it thinks are relevant instead of Hugs performing a search through the list of libraries that it knows to be relevant probably makes things worse because it increases the gap between the programmer's mental model and what actually happens.
Unider Linux it needs not to be specified in which library a symbol is defined. It's enough to say: #include these headers, link in these libraries, and you will get the following functions. (And even libraries need not to be specified by name, as in gtk+ example.) So we can't *require* specifying the exact library for individual functions. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
"Alastair Reid" <reid@cs.utah.edu> wrote,
Marcin just pointed out that you can use dlopen() in a way that it finds symbols without a library name. That should do, I think.
Note that finding a marginally easier way to do the lookup doesn't address my concern that this undirected search will cause maintenance and porting problems by worsening an already bad situation (just which library did Linux/solaris/freebsd/hpux/redhat6.0/redhat7.0/... choose to hide a symbol in).
In fact, I expect that having dlopen do a search through the libraries that it thinks are relevant instead of Hugs performing a search through the list of libraries that it knows to be relevant probably makes things worse because it increases the gap between the programmer's mental model and what actually happens. [..] ps dlopen is not completely portable - Hugs also uses shl_{load,findsym} (HPUX only?) and {LoadLibrary, GetProcAddress} (Win32).
Fair enough, but as I pointed out. Adding a library name to the import declaration also doesn't work in all cases. So we have one ad-hoc, half-backed solution against another. In this case, I would prefer the simpler one (which obviously is not to specify anything, as we can't properly specify it anyway). Unless, we find a third, actually working solution of course. Manuel
The first time I read this syntax, I misinterpreted it. I think my misinterpretation might be interesting so I'll give it below in case anyone else likes it.
foreign library C "<ncurses/curses.h>" "-lcurses" foreign library java "blah" "blah"
foreign import C "curses_function" cf :: Int -> IO Int foreign import java "java.blah.*/foo" jf :: Int -> IO Int
What I first thought this was doing was: 1) The foreign library decls _declare_ two new kinds of external function which the programmer as arbitrarily (and confusingly) chosen to call "C" and "java". 2) The foreign import declarations refer to these new kinds of external function. Those declared as "C" will use <ncurses/curses.h>, etc. Those declared as "java" will use "blah", etc. Of course, this interpretation doesn't quite work because if "C" is just an arbitrary varid with no particular meaning, then we need to say somewhere else that "C" is to use the C calling convention. (We also need to change "C" to "c" if it is going to be a varid.)
A module can have one "foreign defaults" declaration per calling convention, which specifies a string.
foreign defaults ccall "<gtk/gtk.h>: " foreign defaults java "java."
Alastair ps Just as a general remark on language design... User defined defaults are really tempting to add (my component programming language had them briefly and I am under constant pressure to add them back in) but they are kinda messy because: 1) You have to resolve (or forbid) use of multiple defaults. 2) You need a way to override the default. Do you override the whole default (includes, libraries, etc.) at once or only the bits you don't specify. The latter seems to be slightly better than the former because defaults are sorta invisible - the programmer forgets that they're there and so they forget that overriding the default will zap all the default not just the part (libraries say) that they're thinking about at that moment. 3) A small change to code in one place can have unexpected results (and it is hard to detect the consequences using typechecking or from a failed link or whatever). 4) Cutting and pasting some code from one context to another context results in very different behaviour because the defaults don't carry over. [I think every form of default behaviour in the Haskell spec has caused problems like this way out of proportion to the apparent difficulty of specifying the meaning of defaults and the difficulty of implementing it.]
"Alastair Reid" <reid@cs.utah.edu> wrote,
ps Just as a general remark on language design...
User defined defaults are really tempting to add (my component programming language had them briefly and I am under constant pressure to add them back in) but they are kinda messy because:
1) You have to resolve (or forbid) use of multiple defaults.
2) You need a way to override the default. Do you override the whole default (includes, libraries, etc.) at once or only the bits you don't specify.
The latter seems to be slightly better than the former because defaults are sorta invisible - the programmer forgets that they're there and so they forget that overriding the default will zap all the default not just the part (libraries say) that they're thinking about at that moment.
3) A small change to code in one place can have unexpected results (and it is hard to detect the consequences using typechecking or from a failed link or whatever).
4) Cutting and pasting some code from one context to another context results in very different behaviour because the defaults don't carry over.
[I think every form of default behaviour in the Haskell spec has caused problems like this way out of proportion to the apparent difficulty of specifying the meaning of defaults and the difficulty of implementing it.]
In fact, I am tend to agree with you on that point by now. Any handling of defaults will either require some significant machinery (Marcin's proposal) or be a compromise with rough edges (all the other proposals). I am still against cpp, but maybe we can relegate defaults to the tools domain in general (and whoever is fond of cpp, can use cpp). Without defaults, it is still possible to use all features of the FFI. If a user wants the convenience of defaults, she has to use a tool (as with all the other "features" that we considered too heavy for the core FFI). Then, we could use the following general form of foreign imports: foreign import [callconv] [funname] ['unsafe'] varid :: prim_type where funname is calling-convention dependent and in the case of ccall is either merely the name of the imported symbol or an include specification followed by a colon and, then, the imported symbol. So, we have foreign import ccall "gtk_window_new" windowNew :: CInt -> IO (Ptr Window) or foreign import ccall "<gtk/gtk.h>:gtk_window_new" windowNew :: CInt -> IO (Ptr Window) If the <.> are omitted in the include specification, it corresponds to an #include"...". This is simple and largely backwards-compatible. Implementations will support the specification of a library name (in a second string) for some more time, but its use is deprecated. (See the previous emails for why no library spec and why add the include spec to the function name.) Cheers, Manuel
Wed, 21 Feb 2001 15:51:35 -0700, Alastair Reid <reid@cs.utah.edu> pisze:
User defined defaults are really tempting to add (my component programming language had them briefly and I am under constant pressure to add them back in) but they are kinda messy because:
They are sometimes bessy, but it's not that bad in this case. Think about the FFI defaults as special variable definitions which are spelled differently, and proper FFI declarations that use whatever special variables are in scope.
1) You have to resolve (or forbid) use of multiple defaults.
Forbid. You can't have two variables with the same name at the toplevel.
2) You need a way to override the default. Do you override the whole default (includes, libraries, etc.) at once or only the bits you don't specify.
Only the bits you don't specify. You can shadow a name locally (as in SML's 'local' declarations which create a local environment for some definitions).
3) A small change to code in one place can have unexpected results (and it is hard to detect the consequences using typechecking or from a failed link or whatever).
The same can be told about changing a function body. It influences the meaning of all code which uses it.
4) Cutting and pasting some code from one context to another context results in very different behaviour because the defaults don't carry over.
The same can be told about the environment of defined names. You can't cut and paste without considering the visible names anyway. A difference between FFI defaults and Haskell 'default' is that it is used in clearly visible places - foreign declarations - and not at any point the type inference has a specific problem. Specifying include files in each foreign declaration is not an option. We must have a foreign declaration which tells something about the context in which this module and modules importing this module must be compiled. Instead of 'foreign context' or 'foreign language' we may use 'foreign default' and think about it as specifying the defaults which could be specified in foreign declarations if somebody really wanted I.e. using 'foreign default' is not mandatory - you could write everything at each foreign declaration if you wish. I think that the set of attributes is open enough that we must design some keyword/value scheme or such, insead of relying on extending the grammar each time a new kind of attribute is needed. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
participants (4)
-
Alastair Reid -
malcolm-ffi＠cs.york.ac.uk -
Manuel M. T. Chakravarty -
qrczak＠knm.org.pl