February 2001 - FFI - Haskell.org

Typing f.e.d.
by Sven Panne 30 Apr '01

30 Apr '01

I have a small change request regarding foreign export dynamic. Currently the FFI doc says: topdecl : ... .. | 'foreign' 'export' [callconv] 'dynamic' varid :: prim_type -> IO Addr GHC additionally allows: prim_type -> IO Ptr As usual the FFI "looks through" newtypes. But now that we have FunPtr, the following typing makes much more sense: 'foreign' 'export' [callconv] 'dynamic' varid :: prim_type -> IO (FunPtr prim_type) where both prim_types have to be the *same*. We should probably allow the old Addr-typing as well for some time to facilitate the transition, but not the Ptr-typing (bleeding edge people will know what to do :-). The corresponding changes to GHC look easy, so I'd like to commit this if there are no objections. Furthermore, the FFI docs still talk about Addr only, not Ptr/FunPtr. Now that Addr is deprecated, this should be changed, too. Cheers, Sven

5 7

RE: Modified proposal for default decls
by Simon Peyton-Jones 27 Feb '01

27 Feb '01

| I am reasonably convinced by the need for an extensible | attribute-style specification such as Marcin has been proposing. I've been talking about the FFI stuff with Simon and Julian. We aren't (yet, anyway) convinced by the need for anything nearly as elaborate as what is now proposed. Most notably, Malcolm's latest proposal introduces a new form of abstraction (named thing), a bundle of attributes like Gtk or Bzip. Another environment for the compiler to manage! Soon people will want to export these things and import them elsewhere! This way lies madness. What is particularly tanatlising is that to a first approximation none of this is necessary. For example, the native code generator is quite happy without any of the new stuff now proposed. I think we can usefully distinguish three kinds of thing: a) What is necessary to uniquely identify the thing you want to call. In C this is just a single identifier; in Java, it's a fully qualified method name, plus the method type. Other languages doubtless lie between b) What is necessary to allow the compiler to typecheck the foreign import/export. In part, we seek #includes so that in the via-C route the C compiler will compare the call with the prototype, and complain if they don't match. But this is an implementation thing. In principle, the Haskell compiler could consult type info and compare. The important thing is that there's enough info to get from the thing-being-called to the type-of-the-thing-being-called. c) What is necessary for particular compilers, compiling in a particular way, to generate the right code. In particular, GHC compiling via C. GHC can't generate prototypes, because they might be #included. GHC can't omit prototypes unless prototypes are guaranteed to be #included. Hence the motivation to specify #include stuff. Notice that a) is essential b) is desirable, but not implemented at all c) is essential, but extremely language and compiler specific Therefore we propose that a) be part of the FFI standard b) be achievable, if desired, via (a) c) be not part of the FFI standard In particular, we propose to stick to a mild variant of the current FFI story foreign import ccall "gtk:foo" foo :: Int -> IO Int * "ccall" is one of a small number of language specifiers, "java" and ".net" "stdcall" being other possibilities. C is strange in having more than one calling convention; we treat "stdcall" as a second "language". * The "gtk:foo" is a string in language-specific format that uniquely specifies the thing being called. It'll be quite different for Java, when it'll contain the type signature of the method and whether it's a "new" constructor etc. * The "gtk:" part is a concession to (b). It specifies a C package from which this procedure comes. There is then some compiler-specific mechanism for mapping the name of a C package to the location of its header files and .o file. * If the Haskell compiler wants to typecheck the import, it is free to consult the header file, meta data or whatever, based on the foreign package name and the name of the thing being called. No "foreign library", no "foreign default"... You get to repeat everything at every foreign import. Notice that cross-module exports of inlinings are now straightforward: the foreign call carries its package name with it. You may think this is too minimalist, but there are big advantage to simple, minimal designs, even if they can't cope with every single case. Do we really need the elaboration of Malcolm's proposal? Simon

5 8

Modified proposal for default decls
by malcolm-ffi＠cs.york.ac.uk 24 Feb '01

24 Feb '01

I am reasonably convinced by the need for an extensible attribute-style specification such as Marcin has been proposing. However I am also intrigued by Alastair's "interesting" interpretation of one of my earlier suggestions, and it led me to the following proposal, modifying Marcin's design for default decls only slightly: Change foreign_decl ::= 'foreign' what foreign_name attributes varid '::' prim_type | 'foreign' 'default' attributes to foreign_decl ::= 'foreign' what [conid] [attributes] [foreign_name] varid '::' prim_type | 'foreign' 'library' conid attributes I also prefer to change attribute ::= varid | varid string_literal to attribute ::= varid | varid=string_literal The 'default' decl changes to a 'library' decl, and we now introduce a *name* for the set of default features. The conid is an arbitrary name chosen by the user; the library declaration *binds* that name to the given list of features. You can have as many library decls as you like - think of them simply as named collections of features. These names can then be used in a foreign import/export as an abbreviation for those feature specifiers. Additional attributes can be added within the foreign import decl to override ones from the named set, as in Marcin's proposal. So, for example, imagine I wanted to use both Gtk and bzip2 within a single Haskell module. First, set up a bunch of defaults for Gtk: foreign library Gtk header="<gtk/gtk.h>" header="<gtk/glib.h>" dll="libGTK.dll" stdcall unsafe (noting that the order of header files is important), and then another bunch of defaults for bzip2: foreign library Bzip2 header="bzip2.h" ccall When it comes to defining the imported functions, I just use the names I introduced earlier to access the defaults: foreign import Gtk "gtk_text_widget" textWidget :: String -> GtkWidget foreign import Bzip2 bzcat :: String -> Ptr () or, if I want to (say) add an extra header file and override an option for one particular import, I can easily do that: foreign import Gtk safe header="<extra_header.h>" "gtk_example" example :: String -> GtkWidget By the way, this implies that certain single-word features must be invertible - 'unsafe' can be overridden by 'safe', 'nonblocking' can be overridden by 'blocking', or whatever. I think this addresses Alastair's concerns about what happens to defaults when you cut+paste code from one place to another. With this proposal, if you forget to copy the library decl, the compiler will tell you that it doesn't know what defaults you intended, and can even tell you (for instance) "A 'foreign library Bzip2' decl is missing.". The name for a set of default decls is a conid rather than a varid, just to distinguish between the name of the feature *set* and the feature names themselves. They can appear in exactly the same position, and would otherwise have the same form in some cases. Marcin has already listed attributes we already know about, and here is my proposed initial list which coincides to some degree: . safe/unsafe (could rename to memsafe/memunsafe maybe?) . blocking/nonblocking . calling convention (stdcall,ccall,java,...) . header="..." (or maybe call it: include="...") . dll="..." (for interpreters under Windows) . unixso="..." (for interpreters under Unix) Note that 'dynamic' is *not* included in the possible attributes - I think that belongs properly to the function, not to the library. Marcin expressed a similar opinion I think. There is also a suggestion here to try to avoid conditional compilation: that we should have *different* attribute specs for the library name that would be required on different types of machine. Hence the naming of 'dll' and 'unixso' - and maybe there are more possibilities. Of course, this is not a panacea, as we know already from the 'gtk-config --libs' story, but it might go some way to helping. A compiler/interpreter would be free to ignore these specs for shared libraries if it thought it could do a better job by some other means. Regards, Malcolm

2 1

RE: Summary of current change suggestions
by Simon Marlow 24 Feb '01

24 Feb '01

> foreign import ccall "<gtk/gtk.h>:gtk_window_new" > windowNew :: CInt -> IO (Ptr Window) > > If the <.> are omitted in the include specification, it > corresponds to an #include"...". Don't forget you might need multiple includes, eg. <sys/types.h> & <sys/socket.h> for socket(), and ordering is important. Multiple includes could be separated by commas. I agree that adding the include spec to the foreign import declaration is a good idea because of the cross-module inlining issue (I'd forgotten about that). Marcin: I guess hsc2hs should have some magic to insert the include specs automatically :) Cheers, Simon

5 4

RE: Modification to foreign import/export
by Simon Marlow 24 Feb '01

24 Feb '01

> If we want to rule cpp as out of bounds (a very worthwhile > but difficult goal), > then we need to think about how to avoid conditional compilation too. I think the goal is not to avoid the use of cpp, but rather to avoid mentioning it in the spec. In practice, many people will be using cpp (one way or another) when they write a C-library binding. Conditional compilation is a fact of life, I don't think it's worth trying to avoid - but at least there's no need to mention it in the FFI spec. Cheers, Simon

5 7

RE: Summary of current change suggestions
by Simon Marlow 24 Feb '01

24 Feb '01

> On Fri, 23 Feb 2001, Fergus Henderson wrote: > > > OK, now how about things where you need to #define symbols before > > including the header file? > > Choose one: > 1. Write a forwarding header which #defines and #includes. > 2. Use compiler-specific switches to get #defines (-D and > -optc-D in ghc). > 3. Extend the proposed FFI to have define as well as include. > > Unfortunately option 1 doesn't always work well on ghc, because Stg.h > (included into .hc files) indirectly includes headers like <stdlib.h> > *before* user-specified -#include options, so symbols like _BSD_SOURCE > are defined too late to have any effects. This should be fixed. I agree. Would it be fixed by just #including Stg.h after any -#includes, or would that create more problems? Cheers, Simon

3 2

RE: cpp
by Simon Marlow 23 Feb '01

23 Feb '01

> > hsc2hs is in ghc-4.11. > > Did I miss the last 3 releases? I'm still stuck with > something between 4.08.1 > and 4.08.2 waiting for a debian package and freebsd port to > appear for 4.08.2. > :-) 4.11 is the current devel version, which will be released as 5.00. I submitted the 4.08.2 port to the FreeBSD folks just after the release, but they haven't committed it yet. Cheers, Simon

1 0

Summary of current change suggestions
by malcolm-ffi＠cs.york.ac.uk 23 Feb '01

23 Feb '01

Ok FFI folks, I think we're all agreed now on the desirability of tweaking the FFI to permit (a) arbitrary new foreign languages to be added; (b) the "foreignname" specifier to be (potentially) richer, depending on the particular foreign language in question; (c) abstraction of common header/linklib information to a single new sort of decl. For backwards compatibility, I suppose on (b) we are going to have to continue to allow double-string specifiers of the form foreign import "linklib" "cname" hname :: type for some time, although that form would be deprecated from now on? We know mostly what the richer form of the "foreignname" specifier will be for Java, and in any case we can leave that design to SPJ+EM. However, we have also heard one or two half-hearted suggestions to enrich the "foreignname" string for C as well. For instance, Manuel suggested "Gtk: gtk_new_window". Is anyone convinced that this is a good route for C, or should we settle instead on getting (c) right, leaving the "foreignname" untouched for now? As far as (a) is concerned, we have three decisions to make: (1) Do we use quotes around the language name or accept a simple varid? E.g. "java" vs java. As Marcin points out, the names of the languages C and C++ are not varids, so would need to be enclosed in quotes. On the other hand, see decision point (2) below. (2) The actual names for languages or calling conventions. Manuel suggests a change from naming the calling convention to naming the foreign language. This is somewhat attractive. However, I think many languages have a simple C-callable interface and we don't need to know anything more about the actual language. If this is true, I think we should continue to name the calling convention rather than the language. Hence, the names ccall and stdcall might be retained, java can be added. If anyone actually comes up with a C++ binding, a name like cplusplus (for instance) seems good enough to me. This also has the advantage of backwards compatibility, which incidentally would force decision point (1) above towards no-quotes. Incidentally, the intention would _not_ be that all compilers must support all foreign languages. As at present, only the ccall convention is mandatory. Any other named calling convention should be parsed, but if the compiler doesn't support it, a warning (at least) should be issued. In that situation, it is unclear whether the compiler can actually generate sensible code. (At the moment, I think the FFI standard says if your platform doesn't support stdcall, code should default to ccall? Obviously that doesn't scale well to java for instance!) (3) Where do we allow the language specifier? In my opinion, definitely in foreign import and export decls. This is at minimum for backwards compatibility, but looking forward to the day when someone writes FFI code for more than one language in the same module (eek!), it will be needed there. For the same reason, I would argue that it belongs in the new kind of decl required for (c). Now how do we tackle the biggest change, issue (c): abstraction of common header/linklib information to a single new sort of decl? Disregarding syntax for the moment, Marcin has suggested a concrete design that conflicts in some ways with backwards compatibility: * the primary specification is of the language not the calling convention * the calling convention is defined as only one attribute of a language * the calling convention is mandatorily removed from its current position in all functions His proposal also adds many of the features that we want: * names of include files * names of link libraries * decl appears once and scopes over all foreign functions in that module * it is not language specific: the attribute-based style allows new languages to have attributes we haven't even thought of yet. and some features that might be nice additions: * the global decl can be overridden for a single function * using an attribute style directly in the existing foreign import/export decls is potentially cleaner, as it can tidy up current blots like 'unsafe', and admits the possibility of needing further ad-hoc specifications in the future. (Didn't someone have an example that involved blocking/non-blocking thread-safety?) Since I believe backward compatibility is very important, would anyone like to either modify Marcin's proposal, or make a new concrete proposal that is perhaps slightly more conservative but still achieves our goals? I suppose backwards compatibility might not be entirely possible. If it isn't, then we have a choice between making the changes as small as possible (to make porting old code easier), or making them as radical as possible (to ensure that the design is really clean and extensible). Regards, Malcolm

4 11

RE: Summary of current change suggestions
by Simon Marlow 23 Feb '01

23 Feb '01

> > * specifying libraries in the source isn't the right way to > > go. Library names change independently of APIs, and can be > > platform-specific. > > I like the high level goal this is heading towards and was in complete > agreement with implementing it in rules like this until.... > I realised that > all that Hugs sees is the source (i.e., there's no makefile > to put extra > compilation info into) - so if it ain't in the source, where is it? > > But I like the rule... Well... there's no reason why hugs couldn't use something like GHC's packages. In fact, it could use the scheme largely unchanged; but perhaps the package spec should be extended to include a path to source files (the alternative is to overload the path to the interface files and put the sources in the same place if you want to share a package between Hugs & GHC). The package spec format is designed to be used with "read" (i.e. it's Haskell syntax), so Hugs could just read it using its built-in parser. Cheers, Simon

3 2

RE: Summary of current change suggestions
by Simon Marlow 22 Feb '01

22 Feb '01

> On Thu, 22 Feb 2001, Simon Marlow wrote: > > > The package spec format is designed to be used with "read" > (i.e. it's > > Haskell syntax), so Hugs could just read it using its > built-in parser. > > The format should be extensible, i.e. adding future fields should not > break old code which does not use these fields. IMHO each field which > is a list should be allowed to be skipped, defaulting to the > empty list. yep, good point. Simon

1 0