Could FFI support pass-by-value of structs?
It's not usual, but it is allowed to have values of structs passed between functions directly instead of using pointers: /*****/ struct ex { int x; int y; int z; }; ex example_functions (ex p) { (...) } /*****/ Would it be possible to allow that in Haskell FFI by, say, allowing any instance of Storable to be used in a 'foreign' declaration? Like: -- data Ex = (...) instance Storable Ex where sizeOf _ = ... alignment = sizeOf (...) foreign import ccall "example_functions" exampleFunction :: Ex -> IO Ex -- Thanks, Maurício
On Tue, Jun 16, 2009 at 10:48:19AM -0300, Maurício wrote:
It's not usual, but it is allowed to have values of structs passed between functions directly instead of using pointers:
/*****/ struct ex { int x; int y; int z; };
ex example_functions (ex p) { (...) } /*****/
Would it be possible to allow that in Haskell FFI by, say, allowing any instance of Storable to be used in a 'foreign' declaration? Like:
There are a couple problems with this. First, the storage layout for a given C struct may be radically different depending on the back end, even sometimes depending on the operating system. So you can't write a portable Storable instance. The other problem is that the storage layout isn't enough to use structs as arguments as the calling conventions also come into play. Some calling conventions want structs expanded and places on the stack, others want a pointer to some stack allocated data, returning a struct may or may not create a hidden first argument. Now, Haskell compilers already have to tackle calling conventions for basic types, so it is entirely plausable to solve this, the question is, is it worth the effort? anything you write will end up not being portable anyway because it depends on the structure layout. The end result being, structure passing is complicated, so if you need to interface to a library that has pre-defined structeres, use hsc2hs to manually peek and poke into the correct offsets, it will make things easier and be more portable. John -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/
It's not usual, but it is allowed to have values of structs passed between functions directly instead of using pointers: (...)
Would it be possible to allow that in Haskell FFI (...)
There are a couple problems with this. First, the storage layout for a given C struct may be radically different depending on the back end,
When you say struct layout, do you mean the offset of diferent fields? I don't think this would be a problem, as your sugestion of hsc2hs is one of many solutions to that. I'm not sugesting that peek and poke methods of Storage instances should be created automatically, I understand this would not worth the effort. However, isn't just knowing the size and alignment enough to write a generic struct handler that, by using the appropriate calling convention, is going to work with any struct? If not, I agree with you it's really not worth it (as we can use pointers as Felipe sugested. Maurício
On Wed, Jun 24, 2009 at 10:23:29AM -0300, Maurício wrote:
However, isn't just knowing the size and alignment enough to write a generic struct handler that, by using the appropriate calling convention, is going to work with any struct? If not, I agree with you it's really not worth it (as we can use pointers as Felipe sugested.
No, unfortunately it is not. Depending on the actual data types inside the struct, values may be passed in registers, the stack, or split between the two. for instance, the floating point values in a given struct may be passed in SSE registers on x86-64, while the integral parts are passed in normal registers. John -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/
On Wed, 2009-06-24 at 18:48 -0700, John Meacham wrote:
On Wed, Jun 24, 2009 at 10:23:29AM -0300, Maurício wrote:
However, isn't just knowing the size and alignment enough to write a generic struct handler that, by using the appropriate calling convention, is going to work with any struct? If not, I agree with you it's really not worth it (as we can use pointers as Felipe sugested.
No, unfortunately it is not. Depending on the actual data types inside the struct, values may be passed in registers, the stack, or split between the two. for instance, the floating point values in a given struct may be passed in SSE registers on x86-64, while the integral parts are passed in normal registers.
A better approach might be to not use a custom user-defined type with its Storable instance and just use a tuple of FFI-types. Using Maurício's example: /*****/ struct ex { int x; int y; int z; }; ex example_functions (ex p) { (...) } /*****/ foreign import ccall "example_functions" exampleFunction :: (CInt, CInt, CInt) -> IO (CInt, CInt, CInt) So no generic user mechanism, but extend the FFI-marshalable types to include tuples of FFI-types and to pass them using the C struct ABI. I've no idea what happens when using C pack pragmas etc. Duncan
Hello Duncan, Tuesday, June 30, 2009, 12:03:15 AM, you wrote:
struct ex { int x; int y; int z; };
ex example_functions (ex p)
afaik, there is C ABI, that defines how to pass and return parameters of simple types, it's common for all compilers supporting so-called "cdecl" on any architecture (x86, x86-64, ppc, so on). C ABI doesn't define how to pass structures, so we can't generate universal assembler code for this example - it depends on what compiler we want to interact to moreover, even for pointers-to-structures, there is no definitive layout due to packing problem OTOH, we could try to use gcc agreements -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
On Tue, 2009-06-30 at 00:45 +0400, Bulat Ziganshin wrote:
Hello Duncan,
Tuesday, June 30, 2009, 12:03:15 AM, you wrote:
struct ex { int x; int y; int z; };
ex example_functions (ex p)
afaik, there is C ABI, that defines how to pass and return parameters of simple types, it's common for all compilers supporting so-called "cdecl" on any architecture (x86, x86-64, ppc, so on). C ABI doesn't define how to pass structures, so we can't generate universal assembler code for this example - it depends on what compiler we want to interact to
Actually passing structs and unions as arguments or function results is specified by the C ABI. See for example the IA32 ABI: http://www.caldera.com/developers/devspecs/abi386-4.pdf linked from the LSB: http://refspecs.linux-foundation.org/LSB_3.1.0/LSB-Core-IA32/LSB-Core-IA32/n... See the section Function Calling Sequence, 3-9. Specifically: 3-14: Functions Returning Structures or Unions 3-18: Structure and Union Arguments On IA32 structs/unions passed as parameters go by value on the stack. For structs/unions as function results, they are stored into a caller-allocated area on the stack, pointed to by a "hidden" first arg. It's different on each arch, but it's all completely specified.
moreover, even for pointers-to-structures, there is no definitive layout due to packing problem
Actually the padding is also specified by the C ABI. See section 3-3. Compiler pragmas to use packed layout are not ABI conformant (and so typically are only used internally). Duncan
On Tue, Jun 30, 2009 at 01:18:32AM +0100, Duncan Coutts wrote:
On IA32 structs/unions passed as parameters go by value on the stack. For structs/unions as function results, they are stored into a caller-allocated area on the stack, pointed to by a "hidden" first arg.
It's different on each arch, but it's all completely specified.
moreover, even for pointers-to-structures, there is no definitive layout due to packing problem
Actually the padding is also specified by the C ABI. See section 3-3. Compiler pragmas to use packed layout are not ABI conformant (and so typically are only used internally).
Yup. Unfortunately this doesn't help us, since in order to pass structures we need to know more than just the layout (offset and size) of the type, but we need to know the underlying C types as well which a Storable instance doesn't give us. depending on the ABI, struct passing may not be as simple as 'pass on stack' vs 'pass in registers'. it may be something like 'the floating point components get pulled out and put in registers, which the other components get pushed onto the stack'. So, without some compiler magic, like a 'deriving Storable' instance that derived some hidden methods to contain this extra information, or additions to storable, this won't work in general. John -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/
On Tue, 2009-06-30 at 18:59 -0700, John Meacham wrote:
On Tue, Jun 30, 2009 at 01:18:32AM +0100, Duncan Coutts wrote:
On IA32 structs/unions passed as parameters go by value on the stack. For structs/unions as function results, they are stored into a caller-allocated area on the stack, pointed to by a "hidden" first arg.
It's different on each arch, but it's all completely specified.
moreover, even for pointers-to-structures, there is no definitive layout due to packing problem
Actually the padding is also specified by the C ABI. See section 3-3. Compiler pragmas to use packed layout are not ABI conformant (and so typically are only used internally).
Yup. Unfortunately this doesn't help us, since in order to pass structures we need to know more than just the layout (offset and size) of the type, but we need to know the underlying C types as well which a Storable instance doesn't give us. depending on the ABI, struct passing may not be as simple as 'pass on stack' vs 'pass in registers'. it may be something like 'the floating point components get pulled out and put in registers, which the other components get pushed onto the stack'. So, without some compiler magic, like a 'deriving Storable' instance that derived some hidden methods to contain this extra information, or additions to storable, this won't work in general.
Yeah, I'm not suggesting going via Storable (for all those reasons), just extending the FFI to say tuples of FFI types get passed as the corresponding C ABI structs. All the magic to match the current platform C ABI then lives in the compiler. I was only half-serious in suggesting this btw, though as far as I can see it should actually work. It doesn't help with unions of course and it adds complexity to the compiler. Duncan
Yeah, I'm not suggesting going via Storable (for all those reasons), just extending the FFI to say tuples of FFI types get passed as the corresponding C ABI structs. All the magic to match the current platform C ABI then lives in the compiler.
Agree. The tuples idea is far better than my first sugestion on Storable instances.
I was only half-serious in suggesting this btw, though as far as I can see it should actually work. It doesn't help with unions of course and it adds complexity to the compiler.
:( The only known case I have of this beeing important is on GNU GSL, where the complex functions use a complex type defined as a struct (with a size 2 array as the only member). This actually just emulates the 'complex' type of C99, I'm almost sure. Do you imagine an objection on creating a ticket asking for something like CComplex on Foreign.C.Types? Maurício
On Wed, Jul 01, 2009 at 02:22:36PM -0300, Maurício wrote:
Yeah, I'm not suggesting going via Storable (for all those reasons), just extending the FFI to say tuples of FFI types get passed as the corresponding C ABI structs. All the magic to match the current platform C ABI then lives in the compiler.
Agree. The tuples idea is far better than my first sugestion on Storable instances.
I was only half-serious in suggesting this btw, though as far as I can see it should actually work. It doesn't help with unions of course and it adds complexity to the compiler.
:(
The only known case I have of this beeing important is on GNU GSL, where the complex functions use a complex type defined as a struct (with a size 2 array as the only member). This actually just emulates the 'complex' type of C99, I'm almost sure.
I don't know whether it is required that Complex be represented in such a way though.
Do you imagine an objection on creating a ticket asking for something like CComplex on Foreign.C.Types?
No, I would like it. Also add a 'CBool' that maps to the calling convention for _Bool while you are at it. You would need CComplexFloat and CComplexDouble... or perhaps we should just define that 'Complex CFloat' and 'Complex CDouble' are FFI-able types in the obvious way. that is probably better actually. Another FFI related extension I have wanted at times is making user defined types that derive an 'Enum' instance FFI-able, with an 'int' calling convention, translated via toEnum and fromEnum. -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/
Do you imagine an objection on creating a ticket asking for something like CComplex on Foreign.C.Types?
No, I would like it. Also add a 'CBool' that maps to the calling convention for _Bool while you are at it.
Here is a draft. Please comment: Proposal: complement Foreign.C.Types A few more types could be usefull in Foreign.C.Types. These are sugested to be included: * CBool Related to 'bool' (or _Bool) from stdbool.h. * CComplexFloat, CComplexDouble Related to 'float complex' and 'double complex'. Pros: Cons:
On Wed, Jul 01, 2009 at 09:05:06PM -0300, Maurício wrote:
Do you imagine an objection on creating a ticket asking for something like CComplex on Foreign.C.Types?
No, I would like it. Also add a 'CBool' that maps to the calling convention for _Bool while you are at it.
Here is a draft. Please comment:
Proposal: complement Foreign.C.Types
A few more types could be usefull in Foreign.C.Types. These are sugested to be included:
* CBool Related to 'bool' (or _Bool) from stdbool.h.
Good. It appears from the FFI spec that it is possible to define HsBool as '_Bool' and let 'Bool' translate to a 'bool' calling convention. However, it is not guarenteed so a CBool type that is guarenteed to map to _Bool is still good. It would be odd to define CBool as an intergral type though, perhaps we should specify it be defined as newtype CBool = CBool Bool so we can convert between haskell Bool and CBool easily. Though, for consistency with other C types, we may want to
* CComplexFloat, CComplexDouble Related to 'float complex' and 'double complex'.
We need a way to actually examine the values, since 'fromIntegral' won't work. I think the easiest way is to actually reuse Complex which is already in Haskell 98, defining Complex Foo where Foo is a FFI marshable type translates to the complex version of said type. so 'Complex CDouble' translates to the external C type of 'double complex'. We may also want to add 'long double' as a c marshable type too. John -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/
* CComplexFloat, CComplexDouble Related to 'float complex' and 'double complex'.
We need a way to actually examine the values, since 'fromIntegral' won't work. I think the easiest way is to actually reuse Complex which is already in Haskell 98, defining
What do you think of newtype CComplexFloat = CComplexFloat {crealf, cimagf :: CFloat} 'crealf' and 'cimagf' actually belong to the standard, and are supposed to be used to read the real and imaginary part of a float complex. There are also 'creal', 'cimag', 'creall', 'cimagl' that we would use in CComplexDouble and CComplexLDouble.
We may also want to add 'long double' as a c marshable type too.
I think it's already there. Isn't it CLDouble? Maurício
Hello Duncan, Tuesday, June 30, 2009, 4:18:32 AM, you wrote:
Actually passing structs and unions as arguments or function results is specified by the C ABI. See for example the IA32 ABI:
linked from the LSB: http://refspecs.linux-foundation.org/LSB_3.1.0/LSB-Core-IA32/LSB-Core-IA32/n...
sorry, it's "System V ABI, IA32 Supplement" so situation is still the same - afaik, there is no common ABI for all C compilers on x86 platform that describes structures passing and alignment -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
On Thu, 2009-07-02 at 01:26 +0400, Bulat Ziganshin wrote:
Hello Duncan,
Tuesday, June 30, 2009, 4:18:32 AM, you wrote:
Actually passing structs and unions as arguments or function results is specified by the C ABI. See for example the IA32 ABI:
linked from the LSB: http://refspecs.linux-foundation.org/LSB_3.1.0/LSB-Core-IA32/LSB-Core-IA32/n...
sorry, it's "System V ABI, IA32 Supplement"
so situation is still the same - afaik, there is no common ABI for all C compilers on x86 platform that describes structures passing and alignment
You don't need it to be the same between Windows and Unix, it just has to be standard on each platform, which it is. There are really only two ABIs in common use on x86, the System V ABI and the MS one (which apart from the stdcall calling convention only differs in the bitfield layout iirc). But within each platform it most definitely is the same between C compilers[*], that's the whole point. These days there's even a standard C++ ABI. That took ages to standardise, the C one has been around for much much longer. Duncan [*] On Windows gcc can do both normal and MS layout of bitfields.
Hello Duncan, Thursday, July 2, 2009, 2:57:29 AM, you wrote:
You don't need it to be the same between Windows and Unix, it just has to be standard on each platform, which it is. There are really only two ABIs in common use on x86, the System V ABI and the MS one (which apart from the stdcall calling convention only differs in the bitfield layout iirc).
you mean that on windows gcc, msvc and all other C compilers use the same ABI for passing and packing structs? (at least it does noting common to document you shown, it says about SysV ABI, that may be common for all modern unicies - i don't know) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
On Thu, 2009-07-02 at 03:01 +0400, Bulat Ziganshin wrote:
Hello Duncan,
Thursday, July 2, 2009, 2:57:29 AM, you wrote:
You don't need it to be the same between Windows and Unix, it just has to be standard on each platform, which it is. There are really only two ABIs in common use on x86, the System V ABI and the MS one (which apart from the stdcall calling convention only differs in the bitfield layout iirc).
you mean that on windows gcc, msvc and all other C compilers use the same ABI for passing and packing structs?
When using the gcc option -mms-bitfields then yes, gcc and msvc use the same ABI for packing structs. As the name suggests, the only difference otherwise is that MSVC uses a non-standard layout for bitfields, but since it's the de-facto native compiler it kind of defines the native C ABI. So as it happens this isn't relevant for the FFI because it does not support bitfields. Duncan
On Thu, Jul 02, 2009 at 03:01:48AM +0400, Bulat Ziganshin wrote:
Hello Duncan,
Thursday, July 2, 2009, 2:57:29 AM, you wrote:
You don't need it to be the same between Windows and Unix, it just has to be standard on each platform, which it is. There are really only two ABIs in common use on x86, the System V ABI and the MS one (which apart from the stdcall calling convention only differs in the bitfield layout iirc).
you mean that on windows gcc, msvc and all other C compilers use the same ABI for passing and packing structs?
Yes. If you think about it, otherwise it would be impossible to interface with system provided shared libraries (like libc) unless there is a standard. It sometimes varies across different OS's, but for any OS/chip combo there is a single defined C ABI. It is generally called the 'system V' ABI for historical reasons, even though chances are people arn't using it for system V. Usually it is provided by the chip manufacturer and all OS's follow it. Unless they feel like being a PITA, but in that case they have their own standards document that gcc will follow as an option. So, yes. there is always _some_ well defined ABI for the C langauge on a given platform. John -- John Meacham - ⑆repetae.net⑆john⑈ - http://notanumber.net/
Hello John, Tuesday, July 7, 2009, 5:48:10 AM, you wrote:
you mean that on windows gcc, msvc and all other C compilers use the same ABI for passing and packing structs?
Yes. If you think about it, otherwise it would be impossible to interface with system provided shared libraries (like libc) unless there is a standard.
first, i asked about Windows. second, afair Win32 API doesn't have any functions that pass or return entire records rather than pointers. third, i've seen incompatibility problems with packing structs between MSVC (8-byte alignment, afair) and Borland C++ (1-byte) -- Best regards, Bulat mailto:Bulat.Ziganshin@gmail.com
Maurício schrieb:
It's not usual, but it is allowed to have values of structs passed between functions directly instead of using pointers: (...)
Would it be possible to allow that in Haskell FFI (...)
There are a couple problems with this. First, the storage layout for a given C struct may be radically different depending on the back end,
When you say struct layout, do you mean the offset of diferent fields? I don't think this would be a problem, as your sugestion of hsc2hs is one of many solutions to that. I'm not sugesting that peek and poke methods of Storage instances should be created automatically, I understand this would not worth the effort.
However, isn't just knowing the size and alignment enough to write a generic struct handler that, by using the appropriate calling convention, is going to work with any struct? If not, I agree with you it's really not worth it (as we can use pointers as Felipe sugested.
I tried that in http://hackage.haskell.org/package/storable-record/
On Tue, Jun 16, 2009 at 10:48:19AM -0300, Maurício wrote:
/*****/ struct ex { int x; int y; int z; };
ex example_functions (ex p) { (...) } /*****/
You may adopt the approach I used with Hipmunk[1] where there is a wrapper library written in C. For your example you could do something like
void wr_example_functions(ex *p) { *p = example_functions(*p); }
and in you Haskell code you write something like
foreign import ccall unsafe "wrapper.h" wr_example_functions :: Ptr Ex -> IO ()
exampleFunctions :: Ex -> IO Ex exampleFunctions input = with input $ \ptr -> do wr_example_functions ptr peek ptr
The structure is allocated on the stack, so the drawbacks are only another function call and two structure copies, and probably that shouldn't hurt a lot. Note that depending on the C compiler those costs may even get somewhat optimized (e.g. by inlining). [1] http://hackage.haskell.org/package/Hipmunk -- Felipe.
participants (6)
-
Bulat Ziganshin -
Duncan Coutts -
Felipe Lessa -
Henning Thielemann -
John Meacham -
Maurício