Folks, After another teaching-enforced pause, I have continued working at the FFI Report: http://www.cse.unsw.edu.au/~chak/haskell/ffi.ps.gz For C, the report should now be essentially complete (I have included all the libraries). There are, however, still some outstanding issues (see the report and below for the gory details). For all calling conventions other than ccall and stdcall, most is still missing. My plan for driving this further is as follows: * Get all but the most controversial outstanding issues for the language-independent and the C binding fixed until ICFP. * Have a meeting at ICFP to nail any remaining issues in the language-independent and C binding as well as discuss how much we exactly want to do for other calling conventions and how we go about doing it. Could everybody who is coming to ICFP and would like to attend the meeting drop me a note please? Just so that we can fix when to meet. I will be in Florence from Saturday around noon until the following Friday (maybe even Saturday) noon. Cheers, Manuel PS: The source of the document is in cvs.haskell.org and at <http://www.cse.unsw.edu.au/~chak/haskell/ffi.tex>. -=- The Gory Details(TM) -=- I went through all messages on this list that I hadn't responded to yet and either included the suggestions or they are discussed below. "static" keyword ~~~~~~~~~~~~~~~~ Marcin wrote re removing the "static" keyword in C extents,
What the lack of static takes away is the ability to import a function called "dynamic" or "wrapper". I don't think that it's a big limitation considering hundreds of names which are taken away by a Haskell's runtime system.
IHMO, it is one thing for an implementation to do that and another for a language definition.
Anyway, we could call them "_dynamic" and "_wrapper" to indicate that they are magical in some sense - using such names for external function names would be a violation of ISO/ANSI C rules (they are reserved for the C implementation).
*urgh* I think, that's uglier than having "static". So, I'll stick with static unless there are serious counter arguments. Backward compatibility ~~~~~~~~~~~~~~~~~~~~~~ Malcolm raised the issue of backward compatibility. As I understand it we aren't concerned about the definition actually encompassing the syntax and semantics currently implemented in GHC and NHC, but there is an understanding that those system will allow the "old" syntax for a while after they adopt the standard. NHC's current implementation of "foreign export" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
From the discussion, it seems that NHC simply allows the phrase "foreign export" in front of the normal type signature of a function (as opposed to in addition to that type signature). I never had this in mind and it has always been different in GHC and in Sigbjorn's FFI spec. In fact, being able to have
foo :: Num a => a -> a foo = ... foreign export foo :: Int -> Int was always sold as a feature of the system. Further additions to the FFI ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ IMHO, the FFI should really be self-contained and not rely on other non-standardised extensions/libraries. Therefore, I propose some additions. * I think, we should include the `Bits' module (or something similar). It is needed, eg, to handle bindings to C functions that expect bit vectors as arguments. * To implement marshalling for foreign imported pure functions, `unsafePerformIO' is essential. We could add it to the module `Foreign' or `MarshalUtils'. This raises the question of whether `unsafeInterleaveIO' falls in the same category. I think, we should add `unsafePerformIO' and leave `unsafeInterleaveIO' our as it is less important for the FFI. * Why do we only have `Eq' on ForeignPtr, but also `Ord' and `Show' on Ptr? I'll add `Ord' and `Show' to `ForeignPtr' unless their are objections. * Should HsFFI.h include a function like GHC's startupHaskell()? For linking Haskell libraries into foreign code that seems to be necessary. So, I vote for inclusion. Superflous stuff ~~~~~~~~~~~~~~~~ * I am still not convinced that we need `Storable.destruct'. For deallocating special purpose structures that need a deep traversal, shouldn't we just use a custom function? * I am also not really convinced about `MarshalUtils.withMany'. There may be situations, where such a function is handy, but should it really be in the standard libraries? Moreover, it isn't really marshalling specific - it is JAFL (Just Another Function on Lists). Names ~~~~~ * I don't think `CTypesISO' is a good name. Maybe `CTypesLib' as it contains the types from the C library (as opposed to the normal builtin types)? Or put it all into `CTypes' after all - that doesn't mean GHC (or any other compiler) has to put it into a single module, just for the definition. I tend to having one module only. PtrDiff ~~~~~~~ Maybe after all, `PtrDiff' wasn't that bad an idea. To assume that a pointer difference fits into an `Int' (what we do at the moment) is pretty dodgy. Remember that all that H98 requires of an `Int' is that it has >=30 bits. IMHO, this is pretty weak for a general representation of a pointer difference. The annoying thing about our old use of `PtrDiff' was that it made `plusPtr' a pain to use (as we had to cast the offset to a `PtrDiff' first). A simple solution would be plusPtr :: Integral i => Ptr a -> i -> Ptr b I am actually pretty convinced that this is the right way to go. Argument types for foreign functions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ As you can see, I have removed my construction for using `Storable' to specify the types that can be used as arguments to foreign functions. However, I don't agree with the posts claiming that this route would be unworkable. Whenever a type `T' is `Storable', it can be passed to a foreign function in one of two ways: (1) The compiler has special knowledge about the type and just generates the normal (possibly type-dependent) argument passing code. (2) If the compiler has no special knowledge about the type, it `alloca's temporary storage for the type and `poke's the argument into that location. Then, it generates a stub routine in the foreign language that gets a pointer to the temorary storage instead of the actual argument. The stub routine extracts the argument and calls the actual foreign routine. The idea is of course that the compiler (at least if it aspires to generate decent code) has special knowlegde about all the standard types. So, (2) is only the fallback for weird user-defined instances of `Storable'. (+) I like this approach, because it makes the FFI definition nicer. (-) Unfortunately, it is more fuss for the implementation and probably doesn't make a big difference in practice. I have removed it from the report, because "worse is better"[1] (at least that's what some people claim; although it is somewhat against the spirit of Haskell, which tends to put elegance over simplicity of implementation). [1] http://www.jwz.org/doc/worse-is-better.html
NHC's current implementation of "foreign export" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
From the discussion, it seems that NHC simply allows the phrase "foreign export" in front of the normal type signature of a function (as opposed to in addition to that type signature). I never had this in mind and it has always been different in GHC and in Sigbjorn's FFI spec. In fact, being able to have
foo :: Num a => a -> a foo = ...
foreign export foo :: Int -> Int
was always sold as a feature of the system.
If I understand it, the big difference is that GHC will let you write either 2 sigs (one a normal sig, one a foreign export sig) or 1 sig (just the foreign export sig) whereas NHC lets you have just one. I think I prefer to have both because of examples like your foo. (But, Hugs doesn't implement this either because I couldn't quite figure out the typechecker hooks I needed. I regard this (and document it) as a bug.) -- Alastair Reid reid@cs.utah.edu http://www.cs.utah.edu/~reid/
Thanks Manuel for all your work on this document. It looks in pretty good shape to me. This message contains my reactions to the "gory details" from your summary. In another message I'll post some comments on the document itself.
"static" keyword ~~~~~~~~~~~~~~~~ So, I'll stick with static unless there are serious counter arguments.
Fine by me.
Backward compatibility ~~~~~~~~~~~~~~~~~~~~~~ As I understand it we aren't concerned about the definition actually encompassing the syntax and semantics currently implemented.
The actual surface syntax changes are fairly small anyway, so I have no problem with that, and the whole point is to firm up the semantics.
NHC's current implementation of "foreign export" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ being able to have foo :: Num a => a -> a foreign export foo :: Int -> Int was always sold as a feature of the system.
Ok. It wasn't how I read Sigbjorn's original document, but it does seem like a fairly reasonable thing to want to do. Like Alastair says for Hugs though, I'm not entirely sure how to implement it in nhc98, so it may remain as a 'bug' for us too.
Further additions to the FFI ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ IMHO, the FFI should really be self-contained and not rely on other non-standardised extensions/libraries.
Yes, that seems reasonable.
* I think, we should include the `Bits' module. * `unsafePerformIO' is essential. * I'll add `Ord' and `Show' to `ForeignPtr'. * Should HsFFI.h include a function like GHC's startupHaskell()?
Ok to all of these. Nhc98 has haskellInit() rather than startupHaskell(), which just goes to show that if you don't define these things to be standard, implementations will diverge almost immediately.
Superfluous stuff ~~~~~~~~~~~~~~~~ * I am still not convinced that we need `Storable.destruct'.
Agree.
* I am also not really convinced about `MarshalUtils.withMany'.
No opinion.
Names ~~~~~ * I don't think `CTypesISO' is a good name. Maybe `CTypesLib'.
No strong opinion, but if the ISO suffix is inaccurate, Lib would be better.
PtrDiff ~~~~~~~ Maybe after all, `PtrDiff' wasn't that bad an idea. To assume that a pointer difference fits into an `Int' (what we do at the moment) is pretty dodgy.
I agree. However this brings up a wider issue, because the methods of class Storable and functions like mallocBytes also assume that a 30-bit Int will be big enough. Even today some people have machines with gigabytes of RAM, so I don't want the FFI to become inadequate in only a few more years time when multi-gigabytes become common.
Argument types for foreign functions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ As you can see, I have removed my construction for using `Storable' to specify the types that can be used as arguments to foreign functions.
No strong opinion. Regards, Malcolm
Manuel, Here are my comments on the draft FFI Report itself. It looks very good to me, and there are just a few things to mention. Lexical Structure ~~~~~~~~~~~~~~~~~ The lexical syntax adds 'foreign' as a keyword (reservedid). I'm not entirely convinced this is necessary. Certainly in nhc98 we treat 'foreign' as just a specialid, and so like with other specialids you can name a variable 'foreign' if you wish. In other words, is there any real reason to exclude the possibility of module F (foreign, as) where foreign :: X -> Y foreign ... = ... foreign import ccall "something" as :: CInt -> CInt -> CInt which nhc98 currently accepts, but ghc -fglasgow-exts rejects? Standard C Calls ~~~~~~~~~~~~~~~~ The productions fdecl -> 'import' callconv [safety] entity var '::' ftype entity -> " ['static'] [fname] ['&'] ['['lib']'] [cid] " suggest that the entity string must always be present, but could be "". I was wondering if there is any real difficulty in permitting an empty entity string to be omitted altogether? The idea would be to be able to write foreign import ccall sin :: CFloat -> CFloat as at present rather than foreign import ccall "" sin :: CFloat -> CFloat It isn't a big deal, and it might be worth enforcing the literal string quotes just for uniformity, but I thought I'd raise the issue anyway. Int and Word ~~~~~~~~~~~~ You want to drop the assertion that "arithmetic is performed modulo 2^n" for sized Int and Word types, on the grounds that this doesn't hold for Int. But Int is not of fixed size, so how could it require modulo arithmetic! I happen to think the fact that Int is of unspecified size >30 bits, with undefined behaviour on overflow, was something of a mistake in Haskell. Now that we have the opportunity to define a sensible overflow behaviour for fixed size types, I think we should take it. module Ptr ~~~~~~~~~~ The statement "alignPtr yields the next higher address that fulfills the alignment constraint" slightly conflicts with the following one that "this operation is idempotent". I think the first statement really means to say "alignPtr yields an address, the same or next higher, that fulfills the alignment constraint". module StablePtr ~~~~~~~~~~~~~~~~ It seems a little strange that there is an instance of Storable for StablePtr, yet we are forbidden to use the methods of Storable to dereference a StablePtr. In other words, I'm not quite clear on exactly what is being forbidden. C-Specific Marshalling ~~~~~~~~~~~~~~~~~~~~~~ HS_FLOAT_ROUND is called HS_FLOAT_ROUNDS in table 3. Typos ~~~~~ There are a few typos and mis-spellings. Would it be easiest for someone like me to fix these directly via CVS, or is it better to just give you a list privately? I must admit that my experience of using CVS for LaTeX documents was not a happy one, compared to how useful it is for source code. Regards, Malcolm
Malcolm writes:
Now that we have the opportunity to define a sensible overflow behaviour for fixed size types, I think we should take it.
I strongly support Malcolm's proposal. (Which is, of course, to maintain the status quo.) I can think of 2 situations where we would not want this: 1) We want to get errors when overflow occurs. Fix: define new types with this behaviour. The modular behaviour is useful in its own right. But note that it'd be more useful for such types to support Pascal-style range checks since it's much more common for my code to contain integers whose range is limited to 0..999 (say) than for the range boundaries to be powers of 2. 2) You want to use a non-binary representation (e.g., Hugs implements Integer as base 10000 numbers). Fix: None needed - choosing such a representation doesn't prevent you from implementing the specified semantics (though it may be intensely painful or very inefficient). -- Alastair Reid reid@cs.utah.edu http://www.cs.utah.edu/~reid/
Malcolm Wallace <Malcolm.Wallace@cs.york.ac.uk> wrote,
Typos ~~~~~ There are a few typos and mis-spellings. Would it be easiest for someone like me to fix these directly via CVS, or is it better to just give you a list privately? I must admit that my experience of using CVS for LaTeX documents was not a happy one, compared to how useful it is for source code.
Especially with respect to small changes, I think, it is fine to do them directly via CVS. However, to be one the safe side, I propose to always do a cvs update before changing anything and to commit the changes at fairly small intervals. Given that this is LaTeX and we don't have to worry about breaking a build, this shouldn't be a problem. However, if anybody prefers emailing me the corrections for me to enter, I am happy to do so. Manuel PS: More re the technical comments later.
[I'll just respond to a couple of points, the rest I have silently accepted or put on the todo list, or it will be discussed in a later message. This includes Malcolm's first email, too.] Malcolm Wallace <Malcolm.Wallace@cs.york.ac.uk> wrote,
Lexical Structure ~~~~~~~~~~~~~~~~~ The lexical syntax adds 'foreign' as a keyword (reservedid). I'm not entirely convinced this is necessary. Certainly in nhc98 we treat 'foreign' as just a specialid, and so like with other specialids you can name a variable 'foreign' if you wish. In other words, is there any real reason to exclude the possibility of
module F (foreign, as) where
foreign :: X -> Y foreign ... = ...
foreign import ccall "something" as :: CInt -> CInt -> CInt
which nhc98 currently accepts, but ghc -fglasgow-exts rejects?
As I understand H98's distinction between reservedids and specialids, any id that introduces a new grammatical phrase is a reservedid. Otherwise, why is `data' not a specialid? So, I believe it is more inline with H98 to make `foreign' a reservedid.
Standard C Calls ~~~~~~~~~~~~~~~~ The productions
fdecl -> 'import' callconv [safety] entity var '::' ftype entity -> " ['static'] [fname] ['&'] ['['lib']'] [cid] "
suggest that the entity string must always be present, but could be "". I was wondering if there is any real difficulty in permitting an empty entity string to be omitted altogether? The idea would be to be able to write
foreign import ccall sin :: CFloat -> CFloat
as at present rather than
foreign import ccall "" sin :: CFloat -> CFloat
It isn't a big deal, and it might be worth enforcing the literal string quotes just for uniformity, but I thought I'd raise the issue anyway.
Then, we also have to define entity -> [string] at the start of Section 3. I actually considered this, but just went for the simpler syntax. But I would also be happy to allow omitting the "".
Int and Word ~~~~~~~~~~~~ You want to drop the assertion that "arithmetic is performed modulo 2^n" for sized Int and Word types, on the grounds that this doesn't hold for Int. But Int is not of fixed size, so how could it require modulo arithmetic! I happen to think the fact that Int is of unspecified size >30 bits, with undefined behaviour on overflow, was something of a mistake in Haskell. Now that we have the opportunity to define a sensible overflow behaviour for fixed size types, I think we should take it.
Ok - the general opinion here seems clear.
module StablePtr ~~~~~~~~~~~~~~~~ It seems a little strange that there is an instance of Storable for StablePtr, yet we are forbidden to use the methods of Storable to dereference a StablePtr. In other words, I'm not quite clear on exactly what is being forbidden.
The Storable instance allows to store `StablePtr's themselves (ie, the value that represents them) rather than then the value that they refer to - the latter may not be accesses using Storable. Cheers, Manuel
participants (3)
-
Alastair David Reid -
Malcolm Wallace -
Manuel M. T. Chakravarty