ANN: H98 FFI Addendum 1.0, Release Candidate 13
Dear FFI Folks, Despite my claim that RC12 would, modulo errata, become Version 1.0 of the FFI Addendum, I like to propose another (last!) set of changes. The two changes are the following: (1) The addition of a variant of foreign finalizers that take an extra environment argument that facilitates the emulation of closures in C. This adds two functions to `ForeignPtr'. I propose to include this feature, as it has been discussed on this list before, and has only be omitted due to an oversight by me. (Thanks to Peter Gammie for pointing me to the oversight.) (2) The inclusion of John Meacham's `CWString' and `CLString' routines. However, I propose to make the localised versions (aka `CLString') the default and rename the previous (8bit-based) routines to use a `CAString' suffix, where the `A' stands for ASCII. John proposal has been favourably received on this list and it covers an important aspect of string marshalling quite flexibly with a simple interface. Hence, Release Candidate 13 of the H98 FFI Addendum 1.0 is now available from http://www.cse.unsw.edu.au/~chak/haskell/ffi/ Please check sections 5.5 and 6.3, which have been considerably extended. For the second time...I consider Version 1.0 of the FFI Addendum to be completed; no changes except linguistic ones and plain error corrections will be accepted anymore for this version. Cheers, Manuel PS: Both changes have already been discussed with Simon Marlow and Alastair Reid in person. -=- Changes since RC12: * Acks : Added John Meacham * 4.1.5: Bug fix courtesy of Wolfgang Thaller * 5.5 : Added `FinalizerEnvPtr', `newForeignPtrEnv', and `addForeignPtrFinalizerEnv' * 6.3 : Added John Meacham proposal for `wchar_t' support as well localised string marshalling; in particular, this adds `CWString' and `CWStringLen' as well as the `CWString' and the `CAString' family of marshalling routines. In addition, `charIsRepresentable' was added.
On Fri, Oct 31, 2003 at 06:45:41PM +1100, Manuel M T Chakravarty wrote:
(1) The addition of a variant of foreign finalizers that take an extra environment argument that facilitates the emulation of closures in C. This adds two functions to `ForeignPtr'.
Regarding type FinalizerEnvPtr env a = FunPtr (Ptr env -> Ptr a -> IO ()) newForeignPtrEnv :: FinalizerEnvPtr env a -> Ptr env -> Ptr a -> IO (ForeignPtr a) addForeignPtrFinalizerEnv :: FinalizerEnvPtr env a -> Ptr env -> ForeignPtr a -> IO () Doesn't this mean that the env must be either static storage (gaining nothing) or dynamic storage that cannot be reclaimed (which brings us back to where we started)?
I think it's ok for the finalizer function to delete the environment object if it wants isn't it? This isn't always the right thing to do but I think it is in the common case that there is a unique environment object for every finalized object. In the other common case that there is a single environment object for a set of objects (e.g., for all objects of the same 'type'), the finalizer might decrement a reference count on the environment object. Have I missed some subtlety? -- Alastair Reid
Regarding
type FinalizerEnvPtr env a = FunPtr (Ptr env -> Ptr a -> IO ())
newForeignPtrEnv :: FinalizerEnvPtr env a -> Ptr env -> Ptr a -> IO (ForeignPtr a) addForeignPtrFinalizerEnv :: FinalizerEnvPtr env a -> Ptr env -> ForeignPtr a -> IO ()
Doesn't this mean that the env must be either static storage (gaining nothing) or dynamic storage that cannot be reclaimed (which brings us back to where we started)?
On Fri, Oct 31, 2003 at 02:02:08PM +0000, Alastair Reid wrote:
I think it's ok for the finalizer function to delete the environment object if it wants isn't it?
This isn't always the right thing to do but I think it is in the common case that there is a unique environment object for every finalized object.
In the other common case that there is a single environment object for a set of objects (e.g., for all objects of the same 'type'), the finalizer might decrement a reference count on the environment object.
I see.
Alastair Reid <alastair@reid-consulting-uk.ltd.uk> wrote,
I think it's ok for the finalizer function to delete the environment object if it wants isn't it?
This isn't always the right thing to do but I think it is in the common case that there is a unique environment object for every finalized object.
In the other common case that there is a single environment object for a set of objects (e.g., for all objects of the same 'type'), the finalizer might decrement a reference count on the environment object.
That's what I thought, too. Manuel
On Fri, Oct 31, 2003 at 06:45:41PM +1100, Manuel M T Chakravarty wrote:
(2) The inclusion of John Meacham's `CWString' and `CLString' routines. However, I propose to make the localised versions (aka `CLString') the default and rename the previous (8bit-based) routines to use a `CAString' suffix, where the `A' stands for ASCII.
Making the Right Thing the default, though it may cost more, seems appropriate. In the sentence The marshalling takes the current Unicode encoding on the Haskell side into account. (which seems to have been there before), "current" seems wrong, since the Haskell side is constant. How about something like The marshalling converts each Haskell character, representing a Unicode code point, to one or more bytes in a manner determined by the current locale. and dropping the later sentence about the locale. What happens if one attempts to convert a Char that has no encoding in the current locale? It might be worth emphasizing that the Len is the number of bytes rather than Chars. In the part about the single-byte versions, it might be worth tightening the warning to say that these preserve only the first 256 values of Char. (That is the Latin-1 subset, so calling them ASCII seems a misnomer.)
On Fri, Oct 31, 2003 at 12:32:55PM +0000, Ross Paterson wrote:
On Fri, Oct 31, 2003 at 06:45:41PM +1100, Manuel M T Chakravarty wrote:
(2) The inclusion of John Meacham's `CWString' and `CLString' routines. However, I propose to make the localised versions (aka `CLString') the default and rename the previous (8bit-based) routines to use a `CAString' suffix, where the `A' stands for ASCII.
Making the Right Thing the default, though it may cost more, seems appropriate.
In the sentence
The marshalling takes the current Unicode encoding on the Haskell side into account.
(which seems to have been there before), "current" seems wrong, since the Haskell side is constant. How about something like
The marshalling converts each Haskell character, representing a Unicode code point, to one or more bytes in a manner determined by the current locale.
and dropping the later sentence about the locale.
This sounds good to me. I also might reword the paragraph introducing the 8bit versions, as the efficiency reason for using them is less important than the API one. meaning that some C APIs specify that a localized string should be passed, while others explicitly don't use localization and only expect ASCII (or another specific encoding such as utf8) strings and this is most likely what will determine the choice of string marshalers.
What happens if one attempts to convert a Char that has no encoding in the current locale?
my implementation converts unrepresentable characters to '?'. But a case could be made for throwing a CharsetConversion exception of some sort or simply eliding invalid characters. I am not sure what is best, I chose the '?' route because it matches what happens when you don't have a font installed and get a replacement character and is less troublesome for the user. however, it is a lossy transformation (the original character code is lost) so perhaps an exception is better.
It might be worth emphasizing that the Len is the number of bytes rather than Chars.
In the part about the single-byte versions, it might be worth tightening the warning to say that these preserve only the first 256 values of Char. (That is the Latin-1 subset, so calling them ASCII seems a misnomer.)
yeah, this sounds more precise to me too. -- --------------------------------------------------------------------------- John Meacham - California Institute of Technology, Alum. - john@foo.net ---------------------------------------------------------------------------
John Meacham <john@repetae.net> wrote,
On Fri, Oct 31, 2003 at 12:32:55PM +0000, Ross Paterson wrote:
Making the Right Thing the default, though it may cost more, seems appropriate.
In the sentence
The marshalling takes the current Unicode encoding on the Haskell side into account.
(which seems to have been there before), "current" seems wrong, since the Haskell side is constant. How about something like
The marshalling converts each Haskell character, representing a Unicode code point, to one or more bytes in a manner determined by the current locale.
and dropping the later sentence about the locale.
This sounds good to me. I also might reword the paragraph introducing the 8bit versions, as the efficiency reason for using them is less important than the API one. meaning that some C APIs specify that a localized string should be passed, while others explicitly don't use localization and only expect ASCII (or another specific encoding such as utf8) strings and this is most likely what will determine the choice of string marshalers.
True. I changed that.
What happens if one attempts to convert a Char that has no encoding in the current locale?
my implementation converts unrepresentable characters to '?'. But a case could be made for throwing a CharsetConversion exception of some sort or simply eliding invalid characters. I am not sure what is best, I chose the '?' route because it matches what happens when you don't have a font installed and get a replacement character and is less troublesome for the user.
I agree. I now documented to conversion to '?'. Cheers, Manuel
I have put RC 14 at http://www.cse.unsw.edu.au/~chak/haskell/ffi/ including all the feedback on RC13. Please especially have a look at Section 6.3 (Section "CString"), where some of the wording changed. Cheers, Manuel
W liście z sob, 01-11-2003, godz. 02:16, John Meacham pisze:
my implementation converts unrepresentable characters to '?'. But a case could be made for throwing a CharsetConversion exception of some sort or simply eliding invalid characters. I am not sure what is best,
It depends on the application: - When displaying email on screen, it's best to transliterate by removing accents etc. - When communicating with a database, it's best to report an error so configuration can be fixed (perhaps charsets are wrongly set). - When converting HTML, it's best to provide a hook to replace unrepresentable characters with &#NNNN; -- __("< Marcin Kowalczyk \__/ qrczak@knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/
Ross Paterson <ross@soi.city.ac.uk> wrote,
In the sentence
The marshalling takes the current Unicode encoding on the Haskell side into account.
(which seems to have been there before), "current" seems wrong, since the Haskell side is constant. How about something like
The marshalling converts each Haskell character, representing a Unicode code point, to one or more bytes in a manner determined by the current locale.
and dropping the later sentence about the locale.
Done.
It might be worth emphasizing that the Len is the number of bytes rather than Chars.
Done.
In the part about the single-byte versions, it might be worth tightening the warning to say that these preserve only the first 256 values of Char.
Ok.
(That is the Latin-1 subset, so calling them ASCII seems a misnomer.)
Yes, but we can't call it Latin-1 either, because that implies a locale (there is also Latin-2 etc). Manuel
On Sun, Nov 02, 2003 at 09:31:30PM +1100, Manuel M T Chakravarty wrote:
Ross Paterson <ross@soi.city.ac.uk> wrote,
(That is the Latin-1 subset, so calling them ASCII seems a misnomer.)
Yes, but we can't call it Latin-1 either, because that implies a locale (there is also Latin-2 etc).
CByteString? Do you intend to wait until an implementation of this is committed to CVS before finalizing the spec? (I hope so.) Incidentally, the spec says CWChar but the libraries say CWchar. It seems they always disagreed. (If the libs change, maybe CPtrdiff and CFpos should be CPtrDiff and CFPos too.)
Ross Paterson <ross@soi.city.ac.uk> wrote,
On Sun, Nov 02, 2003 at 09:31:30PM +1100, Manuel M T Chakravarty wrote:
Ross Paterson <ross@soi.city.ac.uk> wrote,
(That is the Latin-1 subset, so calling them ASCII seems a misnomer.)
Yes, but we can't call it Latin-1 either, because that implies a locale (there is also Latin-2 etc).
CByteString?
That would be CBString given the current naming scheme. I don't think that this helps much, as one might argue that in CString, we also use bytes on the C side. ASCII may not be entirely correct, but it conveys the right intuition.
Do you intend to wait until an implementation of this is committed to CVS before finalizing the spec? (I hope so.)
We already have a reference implementation (by John Meacham), which is available online. I am not sure whether having it in CVS is going to add much. But we can do this in two steps, (1) another RC announcement at haskell@haskell.org (the last was to the FFI list only). (2) Then, commit after that.
Incidentally, the spec says CWChar but the libraries say CWchar. It seems they always disagreed. (If the libs change, maybe CPtrdiff and CFpos should be CPtrDiff and CFPos too.)
I think, the spec should change. The naming scheme is that we capitalise letters in the Haskell identifier after an underscore in the C identifier; thus, ptrdiff_t -> CPtrdiff sig_atomic_t -> CSigAtomic I will change the spec. Cheers, Manuel
On Wed, Nov 05, 2003 at 07:24:03PM +1100, Manuel M T Chakravarty wrote:
Do you intend to wait until an implementation of this is committed to CVS before finalizing the spec? (I hope so.)
We already have a reference implementation (by John Meacham), which is available online. I am not sure whether having it in CVS is going to add much.
I think it will improve the chances of catching problems in the details, which are a special risk when adding sizeable things late in the process. Looking through that implementation, I note (in relation to Simon's question) that the CWString functions are only provided if the C compiler defines __STDC_ISO_10646__ (so the conversion is fromIntegral). Should the spec say that? If not, and the conversion is possibly locate-dependent, you'd need a counterpart of charIsRepresentable for the wide representation. You have the bit about unrepresentable Chars becoming '?' in the legacy byte string part. I think it belongs in locale-dependent CString part. (I believe the single byte conversion discards all but the lower 8 bits, though there's probably no need to say that.) Other minor things (about old stuff): * I think the description of newCString, etc needs to say that the memory so allocated may be released using MarshalAlloc.free. * CString is said to be NUL-terminated, but presumably CStringLen is not assumed to have a NUL. Perhaps this would be clarified by defining it as (Ptr CChar, Int) instead of (CString, Int) and similarly for CWString.
Ross Paterson <ross@soi.city.ac.uk> wrote,
You have the bit about unrepresentable Chars becoming '?' in the legacy byte string part. I think it belongs in locale-dependent CString part. (I believe the single byte conversion discards all but the lower 8 bits, though there's probably no need to say that.)
Oops. I moved the comment. In fact, I added to the locale-dependent CString part the following text: The translation between Unicode and the encoding of the current locale may be lossy. The function \code{charIsRepresentable} identifies the characters that can be accurately translated; unrepresentable characters are converted to `?'.
Other minor things (about old stuff):
* I think the description of newCString, etc needs to say that the memory so allocated may be released using MarshalAlloc.free.
Good point. I added a sentence to that effect.
* CString is said to be NUL-terminated, but presumably CStringLen is not assumed to have a NUL. Perhaps this would be clarified by defining it as (Ptr CChar, Int) instead of (CString, Int) and similarly for CWString.
Ok, I changed the definition of the type synonyms accordingly. Cheers, Manuel
Manuel M T Chakravarty wrote:
[...] For the second time...I consider Version 1.0 of the FFI Addendum to be completed; no changes except linguistic ones and plain error corrections will be accepted anymore for this version.
Never say never... The signatures for peekElemOff and pokeElemOff in the report are less general than the ones in fptools/libraries/base: peekElemOff :: Storable a => Ptr a -> Int -> IO a vs. peekElemOff :: Storable a => Ptr b -> Int -> IO a (same for pokeElemOff). The latter is more flexible and is more in line with the signatures of plusPtr/minusPtr. Furthermore, changing such a basic function would probably break quite a lot of code, so I propose to change the report, not the libraries. Cheers, S.
[As I was about to send this, it occured to me that maybe you had made a typo and meant to write 'peekByteOff' instead of 'peekElemOff'? This seems less dodgy since peekByteOff is more usually used to access elements of structs where different elements have different types.] I think the report is right and the libraries should be fixed even if a lot of code has to be changed to make the two agree. With the report version: 1) Simple programming errors are readily caught. 2) Less type information needs to be written (obviously at the cost of not catching the type errors in (1).) That is, if you specify the type of the value being peeked, you can infer the type of the pointer and vice-versa. 3) If you want to do something funny, you have to explicitly cast the pointer to make it happen which alerts readers to the fact that you're doing something funny. To be honest, I can't think of many cases where I would want the types you propose. peekElemOff is basically for arrays where all elements have the same type - why would you want to treat them as having different types? -- Alastair Reid
Never say never... The signatures for peekElemOff and pokeElemOff in the report are less general than the ones in fptools/libraries/base:
peekElemOff :: Storable a => Ptr a -> Int -> IO a
vs.
peekElemOff :: Storable a => Ptr b -> Int -> IO a
(same for pokeElemOff). The latter is more flexible and is more in line with the signatures of plusPtr/minusPtr. Furthermore, changing such a basic function would probably break quite a lot of code, so I propose to change the report, not the libraries.
Alastair Reid wrote:
[As I was about to send this, it occured to me that maybe you had made a typo and meant to write 'peekByteOff' instead of 'peekElemOff'? This seems less dodgy since peekByteOff is more usually used to access elements of structs where different elements have different types.]
*aaargl* Of course, sorry for the confusion caused by my typo... I fully agree with your arguments for {peek,poke}ElemOff, but as you already mentioned above, the typical scenario for {peek,poke}ByteOff is access to structs, so the more general type is much more useful. It could be argued that this is a cast via the backdoor, but that would hold for {plus,minus}Ptr, too. Cheers, S.
Sven Panne <Sven.Panne@informatik.uni-muenchen.de> wrote,
Alastair Reid wrote:
[As I was about to send this, it occured to me that maybe you had made a typo and meant to write 'peekByteOff' instead of 'peekElemOff'? This seems less dodgy since peekByteOff is more usually used to access elements of structs where different elements have different types.]
*aaargl* Of course, sorry for the confusion caused by my typo... I fully agree with your arguments for {peek,poke}ElemOff, but as you already mentioned above, the typical scenario for {peek,poke}ByteOff is access to structs, so the more general type is much more useful. It could be argued that this is a cast via the backdoor, but that would hold for {plus,minus}Ptr, too.
I wonder how this discrepancy between the libraries and the report arose. Did I simply make a mistake when writting the report (then, this would qualify as an error in the report anyway)? Or did somebody change the library without discussing the change (or me missing the discussion)? Any ideas? Cheers, Manuel
I wonder how this discrepancy between the libraries and the report arose.
For what it's worth, the library has been that way since June 28, 2001: http://cvs.haskell.org/cgi-bin/cvsweb.cgi/fptools/libraries/base/Foreign/ Storable.hs?rev=1.1&content-type=text/x-cvsweb-markup http://cvs.haskell.org/cgi-bin/cvsweb.cgi/fptools/libraries/base/GHC/ Storable.lhs?rev=1.1&content-type=text/x-cvsweb-markup This seems to be copied from Marcin's QForeign which had the same type from Jan 2, 2001 - March 18, 2001 (i.e., the entire lifetime of the library) http://cvs.sourceforge.net/viewcvs.py/qforeign/qforeign/lib/ QStorable.hsc?rev=1.7&view=markup
Did I simply make a mistake when writting the report (then, this would qualify as an error in the report anyway)?
On this evidence, it looks like it's an error in the report. The type is certainly consistent with the type of plusPtr if you think about implementing peekByteOff like this: peekByteOff p o = peek (plusPtr p o) And, since peekByteOff is primarily for use when peekElemOff doesn't work (i.e., for structs), it makes sense to change the type. -- Alastair Reid
Alastair Reid <alastair@reid-consulting-uk.ltd.uk> wrote,
I wonder how this discrepancy between the libraries and the report arose.
For what it's worth, the library has been that way since June 28, 2001:
http://cvs.haskell.org/cgi-bin/cvsweb.cgi/fptools/libraries/base/Foreign/ Storable.hs?rev=1.1&content-type=text/x-cvsweb-markup
http://cvs.haskell.org/cgi-bin/cvsweb.cgi/fptools/libraries/base/GHC/ Storable.lhs?rev=1.1&content-type=text/x-cvsweb-markup
This seems to be copied from Marcin's QForeign which had the same type from Jan 2, 2001 - March 18, 2001 (i.e., the entire lifetime of the library)
http://cvs.sourceforge.net/viewcvs.py/qforeign/qforeign/lib/ QStorable.hsc?rev=1.7&view=markup
Did I simply make a mistake when writting the report (then, this would qualify as an error in the report anyway)?
On this evidence, it looks like it's an error in the report.
I agree. I am changing the report. Thanks for the forensics. And thanks to Sven for spotting the mismatch. Cheers, Manuel
Hello There's a typo on page 13, section 5.3.. I think a Word34 only has 34 bits, not 64 as stated in the table. Regards -- Adrian Hey
participants (7)
-
Adrian Hey -
Alastair Reid -
John Meacham -
Manuel M T Chakravarty -
Marcin 'Qrczak' Kowalczyk -
Ross Paterson -
Sven Panne