Folks, As promised, I have started on a definition of the Haskell FFI. I started by formalising those aspects of foreign exports and imports that we have recently discussed here. You can access the current text at http://www.cse.unsw.edu.au/~chak/haskell/ffi.ps.gz In addition to general comments, I would particularly welcome your opinion on the FIXME issues. You may note that I have tilted the specification of the calling convention and the related string determining the external entity very much into the direction of the execution platform. In my opinion that makes the whole description much more consistent. In particular, we are now talking about JVM calls rather than Java calls. This fits much better with ccall, wincall, and dotnet. It also makes it much more natural to talk about calling conventions rather than programming languages implemented in terms of these calling conventions. I will add more material (including also a specification of the FFI library) as we go. Cheers, Manuel
As promised, I have started on a definition of the Haskell FFI. I started by formalising those aspects of foreign exports and imports that we have recently discussed here.
This document looks like a very good starting point. * An omission: 'foreign label'. * Perhaps the 'specialid' production should contain all the calling convention identifiers? Regards, Malcolm
malcolm-ffi@cs.york.ac.uk wrote,
This document looks like a very good starting point.
* An omission: 'foreign label'.
Oops.
* Perhaps the 'specialid' production should contain all the calling convention identifiers?
Yes, I had a weird idea here, but realise now that it doesn't work anyway. qrczak@knm.org.pl (Marcin 'Qrczak' Kowalczyk) wrote,
Your syntax puts modifiers between the calling convention and the external id. I agree that it's consistent. Currently ghc accepts unsafe only between the external id and the Haskell id, and dynamic only instead the external id.
Yes, the idea was that this has to be changed in GHC. Sven Panne <Sven.Panne@informatik.uni-muenchen.de> wrote,
* Because of the reason stated above, I think static/dynamic *must* come first after import/export, but the order of unsafe/safe, callconv, and extent should not matter. The var (*not* varid, it was a design flaw IMHO)
I agree.
must be the last thing before the colon, otherwise it tends to drown in the syntax.
Yes.
* I don't understand the last part of section 3.2.1, mentioning the loading of dynamically loaded libs. Is something like dlopen() meant here or linking against a libfoo.so? And the details of how/when this linking should be done are completely obscure to me.
This is just like the library object specification that we have now also. "Simon Peyton-Jones" <simonpj@microsoft.com> wrote,
| > language specific stuff inside the "..." string | > language independent stuff outside | | But static/dynamic probably means different things, depending | on the callconv.
I think it's arguable that static/dynamic should be inside the ext_ent string. Indeed, one might use static/dynamic for ccall, and virtual/non-virtual/static for Java, etc. "Baking in" static/dynamic for all languages may be inappropriate.
Nevertheless, you propose keeping 'mode' outside the ext_ent string. Why? (Apart from backward compat.)
Meanwhile, I think, you are right, we cannot make a clean language-independent static/dynamic distinction.
Also should 'label' be there? Doesn't make sense for Java, does it?
Hmm, not really. So, also into `extent'... "Simon Marlow" <simonmar@microsoft.com> wrote,
A totally minor point, but 'wincall' doesn't feel right. This alternative calling convention has been around since long before windows (it's always been the default calling convention for Pascal, I think).
I'd stick with 'stdcall' because that's what everyone else seems to call it. gcc has a 'stdcall' function attribute, BTW.
Ok - so who prefers `stdcall' and who something else. BTW, I have only included C++ for the sake of completeness into the calling conventions. As nobody is actively working on this - or is there? - I am quite happy to not define anything about it, but the name of the calling convention. Better no definition than an untested one. Cheers, Manuel
"Manuel M. T. Chakravarty" wrote:
[loading dynamic libs problem] This is just like the library object specification that we have now also.
Huh? What? I've never seen a lib object spec... (or completely forgot it %-)
[static/dynamic + callconv] Meanwhile, I think, you are right, we cannot make a clean language-independent static/dynamic distinction.
The callconv already prescribes the syntax of ext_ent and the typing, so it could prescribe the possibilities for mode, too. And static/dynamic makes sense e.g. for cplusplus and jvm at least, with a slight reinterpretation of the actual meaning, but that's not very crucial. I'm against stuffing everything into ext_ent in a string form, which would mean a mini-language within Haskell (TCL is lurking! :-). My proposal: -- allowed for import/export mode : 'static' -- allowed for ccall/stdcall/cplusplus/jvm/..., default then | 'dynamic' -- allowed for ccall/stdcall/cplusplus/jvm/... | 'interface' -- allowed for jvm | ... And for legacy reasons we have to keep dynamic for some time, anyway.
Also should 'label' be there? Doesn't make sense for Java, does it?
Hmm, not really. So, also into `extent'...
But it *makes* sense for some calling conventions, and it the actual translation will need the callconv (name mangling!), so the only change I propose here is a comment: fkind : 'import' | 'export' | 'label' -- allowed for stdcall/ccall/cplusplus Hmmm, looking at the whole foreign beast gives me a slightly uneasy feeling now. There are loads of strongly interconnected side conditions, but I fear we can't avoid this because of the complexity of the whole topic. But at least we should strive for an ext_ent with minimum complexity when we go the "special id" way, like we did in the past. Another extreme route would be: topdecl : 'foreign' string var '::' type with a separate syntax for the string contents. Is there some "standard" or at least widely accepted way of describing a multi-language FFI? We could re-use this here, but I'm not aware of anything like this... Cheers, Sven
Sven Panne <Sven.Panne@informatik.uni-muenchen.de> wrote,
"Manuel M. T. Chakravarty" wrote:
[loading dynamic libs problem] This is just like the library object specification that we have now also.
Huh? What? I've never seen a lib object spec... (or completely forgot it %-)
Like so foreign import ccall "libgtk.so" "gtk_widget_destroy" gtk_widget_destroy :: Addr -> IO ()
[static/dynamic + callconv] Meanwhile, I think, you are right, we cannot make a clean language-independent static/dynamic distinction.
The callconv already prescribes the syntax of ext_ent and the typing, so it could prescribe the possibilities for mode, too. And static/dynamic makes sense e.g. for cplusplus and jvm at least, with a slight reinterpretation of the actual meaning, but that's not very crucial.
I'm against stuffing everything into ext_ent in a string form, which would mean a mini-language within Haskell (TCL is lurking! :-). My proposal:
-- allowed for import/export mode : 'static' -- allowed for ccall/stdcall/cplusplus/jvm/..., default then | 'dynamic' -- allowed for ccall/stdcall/cplusplus/jvm/... | 'interface' -- allowed for jvm | ...
The current idea is to make the syntax of the FFI extension independent of the supported languages. This requires to stuff everything into `extent'. Sure, this will make the `extent' string complex for some languages, but that's inevitable for anything but C anyway. The old design really benefitted from that we basically considered C only. The goal now is to bring the whole thing into better shape for supporting multiple languages and being open for addition of new calling conventions later.
And for legacy reasons we have to keep dynamic for some time, anyway.
Let's forget about the legacy stuff for the FFI definition. The systems have to support the legacy syntax, but that's really of no concern for the definition.
Also should 'label' be there? Doesn't make sense for Java, does it?
Hmm, not really. So, also into `extent'...
But it *makes* sense for some calling conventions, and it the actual translation will need the callconv (name mangling!), so the only change I propose here is a comment:
fkind : 'import' | 'export' | 'label' -- allowed for stdcall/ccall/cplusplus
Hmmm, looking at the whole foreign beast gives me a slightly uneasy feeling now. There are loads of strongly interconnected side conditions, but I fear we can't avoid this because of the complexity of the whole topic. But at least we should strive for an ext_ent with minimum complexity when we go the "special id" way, like we did in the past.
The other design, which I believe is cleaner (and which I understand as the consensus at the HIM meeting), is to get rid of all the side conditions and put all the calling convention dependent stuff into `extent'. The latter is calling convention dependent and complicated (for everything but C) anyway. So, let's at least keep the rest simple. In fact, any system supporting a calling convention like jvm, will have to have a parser for extent strings anyway.
Another extreme route would be:
topdecl : 'foreign' string var '::' type
with a separate syntax for the string contents.
Yes that's going into the right direction, but it makes sense to leave both the calling convention as well as import/export out of the string. The calling convention defines which parser to use in the string. That's good for implementers, because then, the parser can be chosen before parsing starts. The import/export information is needed by the name analysis, which we better leave independent of the whole extent mess. In other words, having the calling convention and import/export seperate facilitates more modular compiler support. Cheers, Manuel
Mon, 07 May 2001 00:14:32 +1000, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
The import/export information is needed by the name analysis, which we better leave independent of the whole extent mess.
Export dynamic introduces a name, export static exports an existing name. For that reason putting this distinction in extent seems strange for me. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
6 May 2001 15:50:16 GMT, Marcin 'Qrczak' Kowalczyk <qrczak@knm.org.pl> pisze:
Export dynamic introduces a name, export static exports an existing name. For that reason putting this distinction in extent seems strange for me.
Moreover, export dynamic and import dynamic don't have fname or cid. foreign import dynamic is a shortcut for importing an automatically generated C function which could be easily written by hand. It fits well into regular imports with a single distinguished magical extent. All parameters are in the type, so it can be called "dynamic" or "curry" or "apply" or whatever, perhaps with special characters to distinguish the name from plain functions. foreign export dynamic is a shortcut for importing an automatically generated nontrivial C function, with argument type not conforming to regular imports, which could not be written by hand. So it's actually an *import*! It could be spelled as import with a magical name - again all parameters are in the type. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
"Manuel M. T. Chakravarty" wrote:
Like so
foreign import ccall "libgtk.so" "gtk_widget_destroy" gtk_widget_destroy :: Addr -> IO ()
I still don't understand the dynamic library stuff related to "fname". Is it a hint for a compiler to link against a given library? Any compiler surely has some kind of option/pragma for this already. What should it mean when there is a corresponding static lib, but not a dynamic one? Most of the time one doesn't really care about this. What should it mean for an interpreter? dlopen-ing? How should the condition "... if the external entity has to be obtained from a dynamically loaded library ..." be implemented, i.e. when does "fname" simply denote the header to #include and when does it denote something related to a DLL? It looks like mixing two unrelated concepts here.
The current idea is to make the syntax of the FFI extension independent of the supported languages. This requires to stuff everything into `extent'. [...]
OK, this makes sense, too.
Let's forget about the legacy stuff for the FFI definition. The systems have to support the legacy syntax, but that's really of no concern for the definition. [...]
I just wanted to make sure that legacy syntax is *possible* with my proposal, not that it should find its way into the FFI spec proper. It's quite easy to design things in such a way that legacy support is almost impossible, so we should keep this in mind.
The other design, which I believe is cleaner (and which I understand as the consensus at the HIM meeting), is to get rid of all the side conditions and put all the calling convention dependent stuff into `extent'. The latter is calling convention dependent and complicated (for everything but C) anyway. So, let's at least keep the rest simple.
Hmmm, but there are still side conditions, e.g. the safe/unsafe distinction only makes sense for imports (and only for those which are not "label" imports), not for exports. This should be reflected in the syntax. And the whole modifier story looks a little bit like overkill. Let's try to be not too general initially. Even when we introduce more modifiers, I'm quite sure that they don't make sense equally in imports and exports. So here's my next try: ------------------------------------------------------------------------ topdecl : 'foreign' impexp [callconv] [extent] var '::' type impexp : 'import' ['unsafe'] | 'export' callconv : 'ccall' -- default | 'stdcall' | 'cplusplus' | 'jvm' | 'dotnet' extent : string -- default: the string rep. of var? ------------------------------------------------------------------------ BTW, we should better rename "extent" to something more unambiguous.
[...] but it makes sense to leave both the calling convention as well as import/export out of the string. The calling convention defines which parser to use in the string. [...] The import/export information is needed by the name analysis, which we better leave independent of the whole extent mess.
In other words, having the calling convention and import/export seperate facilitates more modular compiler support.
But for exactly this reason we should re-think the static/dynamic story. I'm with Marcin here, a "foreign export dynamic" *is* an import, although the resulting function is used for exporting other functions to C land. Any suggestions for the "mode" part of the import's extent? Cheers, Sven
Tue, 17 Apr 2001 18:05:48 +1000, Manuel M. T. Chakravarty <chak@cse.unsw.edu.au> pisze:
Some minor points: In C int can't have 8 bits. It is guaranteed to represent at least -2^15..2^15. Your syntax puts modifiers between the calling convention and the external id. I agree that it's consistent. Currently ghc accepts unsafe only between the external id and the Haskell id, and dynamic only instead the external id. It should not matter that C allows colons in #include. Rules of the #include argument are implementation dependent, and it should not be a problem if a programmer can't name a header file included into Haskell using a colon. He will have to choose a name without a colon. But we could also cut the last colon instead of the first: a colon is certainly not valid in a C identifier. -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
A few comments on this FFI spec: * `foreign label' is missing, but that has already been pointed out. But I'm not sure how this should interact with the different calling conventions, e.g. what should the address of a Java method mean? * I'm not very happy with the fact that static/dynamic are only viewed as modifiers, like unsafe/safe. They almost completely change the meaning of a foreign declaration, including the typing rules. In my view, we have *5* different language constructs: `foreign import', `foreign import dynamic', `foreign export', `foreign export dynamic', and `foreign label'. GC-/concurrency-related modifiers or calling conventions on the other hand do not change the basic meaning of a foreign declaration. * Because of the reason stated above, I think static/dynamic *must* come first after import/export, but the order of unsafe/safe, callconv, and extent should not matter. The var (*not* varid, it was a design flaw IMHO) must be the last thing before the colon, otherwise it tends to drown in the syntax. Another possibility would be a rather strict order on the syntactic entities, but that's not of paramount importance. * Will the calling convention have influence on the typing of a foreign declaration? ccall vs. wincall obviously doesn't, but I'm not sure about the other conventions. I haven't thought very deeply about this yet, but this would be an unfortunate situation. * Some care should be taken to be backwards compatible with the existing FFI, e.g. a plain foreign import sin :: Double -> Double should still be valid, at least for a transition time, and should mean the same thing as foreign import ccall safe static "sin" sin :: Double -> Double in Manuel's syntax, probably yielding a deprecation warning. * IIRC we concluded some time ago that it's not feasible to mirror all strange typing phenomena on the foreign side in the Haskell type, e.g. const modifiers, arrays vs. pointers to elements, etc. But I can't find anything in this proposal to specify these things, e.g. in the extent or something else. * What about automatic prototype generation for ccall/wincall functions? Even if don't require such a thing, this problems related to this topic should be mentioned somewhere. * I don't understand the last part of section 3.2.1, mentioning the loading of dynamically loaded libs. Is something like dlopen() meant here or linking against a libfoo.so? And the details of how/when this linking should be done are completely obscure to me. * How will we differentiate between virtual vs. non-virtual methods in the cplusplus calling convention? What about class methods vs. instance methods? Will these distinctions have influence on the typing rules? If yes, coding this into the extent would be wrong. * The JVM has *4* different instructions for method invocation (invokeinterface, invokespecial, invokestatic, and invokevirtual), so our static/dynamic distinction seems a bit inflexible here. And the introduction of the "new" prefix looks like a hack, which doesn't even handle all situations. * The JVM instructions mentioned above can be categorized into two kinds: The first kind expects an object reference in addition to the actual arguments (invokeinterface, invokespecial, invokevirtual), the second kind doesn't (invokestatic). When we export a Haskell function to the JVM, how do we specify this distinction? static vs. dynamic? This means something slightly different in the case of ccall/wincall. What about typing? * How are C++/JVM exceptions handled? I fear that almost no real C++/Java code will be useable from the Haskell side if there is no way to catch these beasts or throw them. I hope I didn't sound too negative, that's not what was intended. It's only a mirror of the fact that there are a lot of loose ends in the spec and a multi-language FFI in a strongly type language is a rather ambitious (but very cool :-) project. Thanks to Manuel for his effort! Cheers, Sven
There is now an updated version of the document at http://www.cse.unsw.edu.au/~chak/haskell/{ffi.ps.gz,ffi.tex} The LaTeX file is to make citing from the document easier (as suggested by Fergus). To compile it, you need a style file which I first have to polish a little more before I put it on the net (will all happen in due course). The new version includes the feedback so far. The main change is that label, static, and dynamic are now part of `extent' for `ccall' and `stdcall'. It still has many gaps, but I think it is important to regularly post updates even if the delta is not very big. Cheers, Manuel
Sorry for lots of small messages. 'label' can be spelled '&'. It's IMHO more intuitive, and reminds that we get a pointer in Haskell and not the dereferenced value. foreign import "&stdscr" stdScr :: Ptr Window -- __("< Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/ \__/ ^^ SYGNATURA ZASTÊPCZA QRCZAK
participants (5)
-
malcolm-ffi＠cs.york.ac.uk -
Manuel M. T. Chakravarty -
Marcin 'Qrczak' Kowalczyk -
qrczak＠knm.org.pl
-
Sven Panne