FFI: c/c++ struct on stack as an argument or return value

Yuras Shumovich

14 Mar 2014 14 Mar '14

7:50 a.m.

Hi, Right now ghc's FFI doesn't support c/c++ structures. Whenever we have foreign function that accepts or returns struct by value, we have to create wrapper that accepts or returns pointer to struct. It is inconvenient, but actually not a big deal. But there is no easy workaround when you want to export haskell function to use it with c/c++ API that requires structures to be passed by value (Usually it is a callback in c/c++ API. You can't change it's signature, and if it doesn't provide some kind of "void* userdata", then you are stuck.) I'm interested in fixing that. I'm going to start with 'foreign import "wrapper" ...' stuff. Calling conventions for passing c/c++ structures by value are pretty tricky and platform/compiler specific. So initially I'll use libffi for that (it will work when USE_LIBFFI_FOR_ADJUSTORS is defined, see rts/Adjustor.c). It will allow me to explore design space without bothering about low level implementation details. Later it could be implemented for native (non-libffi) adjustors. Is anybody interested it that? I appreciate any comments/ideas. Right now I don't have clear design. It would be nice to support plain haskell data types that are 1) not recursive, 2) has one constructor and 3) contains only c/c++ types. But it doesn't work with c/c++ unions. Any ideas are welcome. An example how to use libffi with structures: http://www.atmark-techno.com/~yashi/libffi.html#Structures Thanks, Yuras

Show replies by date

Edward Kmett

14 Mar 14 Mar

9:08 a.m.

I spent some time hacking around on this from a library perspective when I had to interoperate with a bunch of Objective C on a 64-bit mac as many of the core library functions you need to FFI out to pass around pairs of Int32s as a struct small enough by the x64 ABI to get shoehorned into one register, and as I was programmatically cloning Objective C APIs via template haskell I couldn't use the usual clunky C shims. What I was doing was just using libffi with a lot of work to cache the results of ffi_prep_cif for each signature. It worked reasonably well for my purposes, but my need for it vanished and I abandoned the code in the middle of refactoring it for grander things. So if nothing else, you can at least take this as a vote of confidence that your idea isn't crazy. =) I'd also be happy to answer questions if you get stuck or need help. -Edward On Fri, Mar 14, 2014 at 7:50 AM, Yuras Shumovich wrote:

...

Hi,

Right now ghc's FFI doesn't support c/c++ structures.

Whenever we have foreign function that accepts or returns struct by value, we have to create wrapper that accepts or returns pointer to struct. It is inconvenient, but actually not a big deal.

But there is no easy workaround when you want to export haskell function to use it with c/c++ API that requires structures to be passed by value (Usually it is a callback in c/c++ API. You can't change it's signature, and if it doesn't provide some kind of "void* userdata", then you are stuck.)

I'm interested in fixing that. I'm going to start with 'foreign import "wrapper" ...' stuff.

Calling conventions for passing c/c++ structures by value are pretty tricky and platform/compiler specific. So initially I'll use libffi for that (it will work when USE_LIBFFI_FOR_ADJUSTORS is defined, see rts/Adjustor.c). It will allow me to explore design space without bothering about low level implementation details. Later it could be implemented for native (non-libffi) adjustors.

Is anybody interested it that? I appreciate any comments/ideas.

Right now I don't have clear design. It would be nice to support plain haskell data types that are 1) not recursive, 2) has one constructor and 3) contains only c/c++ types. But it doesn't work with c/c++ unions. Any ideas are welcome.

An example how to use libffi with structures: http://www.atmark-techno.com/~yashi/libffi.html#Structures

Thanks, Yuras

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Yuras Shumovich

2 p.m.

On Fri, 2014-03-14 at 09:08 -0400, Edward Kmett wrote:

...

I spent some time hacking around on this from a library perspective when I had to interoperate with a bunch of Objective C on a 64-bit mac as many of the core library functions you need to FFI out to pass around pairs of Int32s as a struct small enough by the x64 ABI to get shoehorned into one register, and as I was programmatically cloning Objective C APIs via template haskell I couldn't use the usual clunky C shims.

Was it related to language-c-inline package?

...

So if nothing else, you can at least take this as a vote of confidence that your idea isn't crazy. =)

I'd also be happy to answer questions if you get stuck or need help.

Thank you, Edward Since there is at least one person how is interested in, I'll start asking questions. Please let me know when I become too noisy :) For now I'm focused on desugaring phase. Right now type Fn = CInt -> CInt -> IO () foreign import ccall "wrapper" f :: Fn -> IO (FunPtr Fn) is desugared into f :: Fn -> IO (FunPtr Fn) f hsFunc = do sPtr <- newStablePtr hsFunc createAdjustor sPtr staticWrapper ... Here staticWrapper -- address of C function. It will dereference the sPtr, cast to StgClosure* and call with appropriate arguments. All the arguments are primitive C types (int, char, pointer, etc), so it is easy to convert them to corresponding haskell types via rts_mkInt, rts_mkChar etc. But I want to allow argument to be C structs. data CStruct { i :: CInt, j :: CInt } type Fn = CStruct -> IO () foreign import ccall "wrapper" f :: Fn -> IO (FunPtr Fn) Looks like it is impossible to instantiate CStruct from C function. Is it true? Is it easy to add such functionality? The only solution I see is to flatten CStruct before creating StablePtr: f :: Fn -> IO (FunPtr Fn) f hsFunc = do sPtr <- newStablePtr $ \i j -> hsFunc (CStruct i j) createAdjustor sPtr staticWrapper ... Does it make sense? It will add performance overhead because of additional indirection. Better ideas are welcome. Thanks, Yuras

...

-Edward

On Fri, Mar 14, 2014 at 7:50 AM, Yuras Shumovich wrote:

...
Hi,

Right now ghc's FFI doesn't support c/c++ structures.

Whenever we have foreign function that accepts or returns struct by value, we have to create wrapper that accepts or returns pointer to struct. It is inconvenient, but actually not a big deal.

But there is no easy workaround when you want to export haskell function to use it with c/c++ API that requires structures to be passed by value (Usually it is a callback in c/c++ API. You can't change it's signature, and if it doesn't provide some kind of "void* userdata", then you are stuck.)

I'm interested in fixing that. I'm going to start with 'foreign import "wrapper" ...' stuff.

Calling conventions for passing c/c++ structures by value are pretty tricky and platform/compiler specific. So initially I'll use libffi for that (it will work when USE_LIBFFI_FOR_ADJUSTORS is defined, see rts/Adjustor.c). It will allow me to explore design space without bothering about low level implementation details. Later it could be implemented for native (non-libffi) adjustors.

Is anybody interested it that? I appreciate any comments/ideas.

Right now I don't have clear design. It would be nice to support plain haskell data types that are 1) not recursive, 2) has one constructor and 3) contains only c/c++ types. But it doesn't work with c/c++ unions. Any ideas are welcome.

An example how to use libffi with structures: http://www.atmark-techno.com/~yashi/libffi.html#Structures

Thanks, Yuras

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Edward Kmett

3:01 p.m.

On Fri, Mar 14, 2014 at 2:00 PM, Yuras Shumovich wrote:

...

On Fri, 2014-03-14 at 09:08 -0400, Edward Kmett wrote:

...
I spent some time hacking around on this from a library perspective when I had to interoperate with a bunch of Objective C on a 64-bit mac as many of the core library functions you need to FFI out to pass around pairs of Int32s as a struct small enough by the x64 ABI to get shoehorned into one register, and as I was programmatically cloning Objective C APIs via template haskell I couldn't use the usual clunky C shims.

Was it related to language-c-inline package?

It was an exercise in serial yak shaving brought about by writing a realtime GPU-based Metropolis light transport raytracer... er.. nevermind. =)

...

So if nothing else, you can at least take this as a vote of confidence that

...
your idea isn't crazy. =)

I'd also be happy to answer questions if you get stuck or need help.

Thank you, Edward

Since there is at least one person how is interested in, I'll start asking questions. Please let me know when I become too noisy :)

For now I'm focused on desugaring phase. Right now

type Fn = CInt -> CInt -> IO () foreign import ccall "wrapper" f :: Fn -> IO (FunPtr Fn)

is desugared into

f :: Fn -> IO (FunPtr Fn) f hsFunc = do sPtr <- newStablePtr hsFunc createAdjustor sPtr staticWrapper ...

Here staticWrapper -- address of C function. It will dereference the sPtr, cast to StgClosure* and call with appropriate arguments. All the arguments are primitive C types (int, char, pointer, etc), so it is easy to convert them to corresponding haskell types via rts_mkInt, rts_mkChar etc.

But I want to allow argument to be C structs.

data CStruct { i :: CInt, j :: CInt } type Fn = CStruct -> IO () foreign import ccall "wrapper" f :: Fn -> IO (FunPtr Fn)

Looks like it is impossible to instantiate CStruct from C function. Is it true? Is it easy to add such functionality?

The only solution I see is to flatten CStruct before creating StablePtr:

f :: Fn -> IO (FunPtr Fn) f hsFunc = do sPtr <- newStablePtr $ \i j -> hsFunc (CStruct i j) createAdjustor sPtr staticWrapper ...

Does it make sense? It will add performance overhead because of additional indirection. Better ideas are welcome.

Not sure. This is a much lower level (and more correct .. and likely faster) approach than I was taking. I'd just built all my functions in a way that would cache the resulting ffi_prep_cif for each signature using typeclass magic. I had to do some allocations on each ffi_call though as well for struct avalues though, so I'm guessing you'd have to do at least that much.

...

Thanks, Yuras

...
-Edward

On Fri, Mar 14, 2014 at 7:50 AM, Yuras Shumovich
...
Hi,

Right now ghc's FFI doesn't support c/c++ structures.

Whenever we have foreign function that accepts or returns struct by value, we have to create wrapper that accepts or returns pointer to struct. It is inconvenient, but actually not a big deal.

But there is no easy workaround when you want to export haskell

function

...
...
to use it with c/c++ API that requires structures to be passed by value (Usually it is a callback in c/c++ API. You can't change it's signature, and if it doesn't provide some kind of "void* userdata", then you are stuck.)

I'm interested in fixing that. I'm going to start with 'foreign import "wrapper" ...' stuff.

Calling conventions for passing c/c++ structures by value are pretty tricky and platform/compiler specific. So initially I'll use libffi for that (it will work when USE_LIBFFI_FOR_ADJUSTORS is defined, see rts/Adjustor.c). It will allow me to explore design space without bothering about low level implementation details. Later it could be implemented for native (non-libffi) adjustors.

Is anybody interested it that? I appreciate any comments/ideas.

Right now I don't have clear design. It would be nice to support plain haskell data types that are 1) not recursive, 2) has one constructor and 3) contains only c/c++ types. But it doesn't work with c/c++ unions. Any ideas are welcome.

An example how to use libffi with structures: http://www.atmark-techno.com/~yashi/libffi.html#Structures

Thanks, Yuras

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Manuel M T Chakravarty

15 Mar 15 Mar

midnight

Yuras, I’m not convinced that the compiler is the right place for this kind of functionality. In fact, when we designed the Haskell FFI, we explicit decided against what you propose. There are a few reasons for this. Firstly, compilers are complex beasts, and secondly, it takes a long time until a change in the compiler goes into production. Hence, as a general rule, it is advisable to move complexity from the compiler into libraries as this reduces compiler complexity. Libraries are less complex and changes can be rolled out much more quickly (it’s essentially a Hackage upload versus waiting for the next GHC and Haskell Platform release). Thirdly, we have got the Haskell standard for a reason and modifying the compiler implies a language extension. The design goal for the Haskell FFI was to provide the absolute minimum as part of the language and compiler, and to layer additional conveniences on top of that in the form of libraries and tools. Have you considered the library or tool route? Manuel Yuras Shumovich :

...

Hi,

Right now ghc's FFI doesn't support c/c++ structures.

Whenever we have foreign function that accepts or returns struct by value, we have to create wrapper that accepts or returns pointer to struct. It is inconvenient, but actually not a big deal.

But there is no easy workaround when you want to export haskell function to use it with c/c++ API that requires structures to be passed by value (Usually it is a callback in c/c++ API. You can't change it's signature, and if it doesn't provide some kind of "void* userdata", then you are stuck.)

I'm interested in fixing that. I'm going to start with 'foreign import "wrapper" ...' stuff.

Calling conventions for passing c/c++ structures by value are pretty tricky and platform/compiler specific. So initially I'll use libffi for that (it will work when USE_LIBFFI_FOR_ADJUSTORS is defined, see rts/Adjustor.c). It will allow me to explore design space without bothering about low level implementation details. Later it could be implemented for native (non-libffi) adjustors.

Is anybody interested it that? I appreciate any comments/ideas.

Right now I don't have clear design. It would be nice to support plain haskell data types that are 1) not recursive, 2) has one constructor and 3) contains only c/c++ types. But it doesn't work with c/c++ unions. Any ideas are welcome.

An example how to use libffi with structures: http://www.atmark-techno.com/~yashi/libffi.html#Structures

Thanks, Yuras

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Carter Schonwald

12:17 a.m.

indeed, its very very easy to do storable instances that correspond to the struct type you want, the ``with`` function in http://hackage.haskell.org/package/base-4.6.0.1/docs/Foreign-Marshal-Utils.h... actually gets you most of the way there! On Sat, Mar 15, 2014 at 12:00 AM, Manuel M T Chakravarty < chak@cse.unsw.edu.au> wrote:

...

Yuras,

I’m not convinced that the compiler is the right place for this kind of functionality. In fact, when we designed the Haskell FFI, we explicit decided against what you propose. There are a few reasons for this.

Firstly, compilers are complex beasts, and secondly, it takes a long time until a change in the compiler goes into production. Hence, as a general rule, it is advisable to move complexity from the compiler into libraries as this reduces compiler complexity. Libraries are less complex and changes can be rolled out much more quickly (it’s essentially a Hackage upload versus waiting for the next GHC and Haskell Platform release).

Thirdly, we have got the Haskell standard for a reason and modifying the compiler implies a language extension.

The design goal for the Haskell FFI was to provide the absolute minimum as part of the language and compiler, and to layer additional conveniences on top of that in the form of libraries and tools.

Have you considered the library or tool route?

Manuel

Yuras Shumovich :

...
Hi,

Right now ghc's FFI doesn't support c/c++ structures.

Whenever we have foreign function that accepts or returns struct by value, we have to create wrapper that accepts or returns pointer to struct. It is inconvenient, but actually not a big deal.

But there is no easy workaround when you want to export haskell function to use it with c/c++ API that requires structures to be passed by value (Usually it is a callback in c/c++ API. You can't change it's signature, and if it doesn't provide some kind of "void* userdata", then you are stuck.)

I'm interested in fixing that. I'm going to start with 'foreign import "wrapper" ...' stuff.

Calling conventions for passing c/c++ structures by value are pretty tricky and platform/compiler specific. So initially I'll use libffi for that (it will work when USE_LIBFFI_FOR_ADJUSTORS is defined, see rts/Adjustor.c). It will allow me to explore design space without bothering about low level implementation details. Later it could be implemented for native (non-libffi) adjustors.

Is anybody interested it that? I appreciate any comments/ideas.

Right now I don't have clear design. It would be nice to support plain haskell data types that are 1) not recursive, 2) has one constructor and 3) contains only c/c++ types. But it doesn't work with c/c++ unions. Any ideas are welcome.

An example how to use libffi with structures: http://www.atmark-techno.com/~yashi/libffi.html#Structures

Thanks, Yuras

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Edward Kmett

12:33 a.m.

I don't care enough to fight and try to win the battle, but I just want to point out that Storable structs are far more brittle and platform dependent than borrowing the already correct platform logic for struct passing from libffi. I do think the existing FFI extension made the right call under the 32 bit ABIs that were in use at the time it was defined. That said, with 64-bit ABIs saying that 2 32-bit ints should be passed in a single 64 bit register, you wind up with large chunks of third party APIs we just can't call out to directly any more, requiring many one-off manual C shims. -Edward On Sat, Mar 15, 2014 at 12:17 AM, Carter Schonwald < carter.schonwald@gmail.com> wrote:

...

indeed, its very very easy to do storable instances that correspond to the struct type you want,

the ``with`` function in http://hackage.haskell.org/package/base-4.6.0.1/docs/Foreign-Marshal-Utils.h... actually gets you most of the way there!

On Sat, Mar 15, 2014 at 12:00 AM, Manuel M T Chakravarty < chak@cse.unsw.edu.au> wrote:

...
Yuras,

I’m not convinced that the compiler is the right place for this kind of functionality. In fact, when we designed the Haskell FFI, we explicit decided against what you propose. There are a few reasons for this.

Firstly, compilers are complex beasts, and secondly, it takes a long time until a change in the compiler goes into production. Hence, as a general rule, it is advisable to move complexity from the compiler into libraries as this reduces compiler complexity. Libraries are less complex and changes can be rolled out much more quickly (it’s essentially a Hackage upload versus waiting for the next GHC and Haskell Platform release).

Thirdly, we have got the Haskell standard for a reason and modifying the compiler implies a language extension.

The design goal for the Haskell FFI was to provide the absolute minimum as part of the language and compiler, and to layer additional conveniences on top of that in the form of libraries and tools.

Have you considered the library or tool route?

Manuel

Yuras Shumovich :

...
Hi,

Right now ghc's FFI doesn't support c/c++ structures.

Whenever we have foreign function that accepts or returns struct by value, we have to create wrapper that accepts or returns pointer to struct. It is inconvenient, but actually not a big deal.

But there is no easy workaround when you want to export haskell function to use it with c/c++ API that requires structures to be passed by value (Usually it is a callback in c/c++ API. You can't change it's signature, and if it doesn't provide some kind of "void* userdata", then you are stuck.)

I'm interested in fixing that. I'm going to start with 'foreign import "wrapper" ...' stuff.

Calling conventions for passing c/c++ structures by value are pretty tricky and platform/compiler specific. So initially I'll use libffi for that (it will work when USE_LIBFFI_FOR_ADJUSTORS is defined, see rts/Adjustor.c). It will allow me to explore design space without bothering about low level implementation details. Later it could be implemented for native (non-libffi) adjustors.

Is anybody interested it that? I appreciate any comments/ideas.

Right now I don't have clear design. It would be nice to support plain haskell data types that are 1) not recursive, 2) has one constructor and 3) contains only c/c++ types. But it doesn't work with c/c++ unions. Any ideas are welcome.

An example how to use libffi with structures: http://www.atmark-techno.com/~yashi/libffi.html#Structures

Thanks, Yuras

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Carter Schonwald

12:37 a.m.

I'm not opposing that, in fact, theres a GHC ticket discussing some stuff related to this (related to complex numbers). i think the crux of Manuel's point is mainly that any good proposal has to at least give a roadmap to support on all the various platforms etc etc On Sat, Mar 15, 2014 at 12:33 AM, Edward Kmett wrote:

...

I don't care enough to fight and try to win the battle, but I just want to point out that Storable structs are far more brittle and platform dependent than borrowing the already correct platform logic for struct passing from libffi.

I do think the existing FFI extension made the right call under the 32 bit ABIs that were in use at the time it was defined. That said, with 64-bit ABIs saying that 2 32-bit ints should be passed in a single 64 bit register, you wind up with large chunks of third party APIs we just can't call out to directly any more, requiring many one-off manual C shims.

-Edward

On Sat, Mar 15, 2014 at 12:17 AM, Carter Schonwald < carter.schonwald@gmail.com> wrote:

...
indeed, its very very easy to do storable instances that correspond to the struct type you want,

the ``with`` function in http://hackage.haskell.org/package/base-4.6.0.1/docs/Foreign-Marshal-Utils.h... actually gets you most of the way there!

On Sat, Mar 15, 2014 at 12:00 AM, Manuel M T Chakravarty < chak@cse.unsw.edu.au> wrote:

...
Yuras,

I’m not convinced that the compiler is the right place for this kind of functionality. In fact, when we designed the Haskell FFI, we explicit decided against what you propose. There are a few reasons for this.

Firstly, compilers are complex beasts, and secondly, it takes a long time until a change in the compiler goes into production. Hence, as a general rule, it is advisable to move complexity from the compiler into libraries as this reduces compiler complexity. Libraries are less complex and changes can be rolled out much more quickly (it’s essentially a Hackage upload versus waiting for the next GHC and Haskell Platform release).

Thirdly, we have got the Haskell standard for a reason and modifying the compiler implies a language extension.

The design goal for the Haskell FFI was to provide the absolute minimum as part of the language and compiler, and to layer additional conveniences on top of that in the form of libraries and tools.

Have you considered the library or tool route?

Manuel

Yuras Shumovich :

...
Hi,

Right now ghc's FFI doesn't support c/c++ structures.

Whenever we have foreign function that accepts or returns struct by value, we have to create wrapper that accepts or returns pointer to struct. It is inconvenient, but actually not a big deal.

But there is no easy workaround when you want to export haskell function to use it with c/c++ API that requires structures to be passed by value (Usually it is a callback in c/c++ API. You can't change it's signature, and if it doesn't provide some kind of "void* userdata", then you are stuck.)

I'm interested in fixing that. I'm going to start with 'foreign import "wrapper" ...' stuff.

Calling conventions for passing c/c++ structures by value are pretty tricky and platform/compiler specific. So initially I'll use libffi for that (it will work when USE_LIBFFI_FOR_ADJUSTORS is defined, see rts/Adjustor.c). It will allow me to explore design space without bothering about low level implementation details. Later it could be implemented for native (non-libffi) adjustors.

Is anybody interested it that? I appreciate any comments/ideas.

Right now I don't have clear design. It would be nice to support plain haskell data types that are 1) not recursive, 2) has one constructor and 3) contains only c/c++ types. But it doesn't work with c/c++ unions. Any ideas are welcome.

An example how to use libffi with structures: http://www.atmark-techno.com/~yashi/libffi.html#Structures

Thanks, Yuras

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Yuras Shumovich

8:51 a.m.

Manuel, I think the compiler is the right place. It is impossible to have efficient implementation in a library. For dynamic wrapper (foreign import "wrapper" stuff) ghc generates piece of executable code at runtime. There are native implementations for a number of platforms, and libffi is used as a fall back for other platforms (see rts/Adjustor.c). AFAIK it is done that way because libffi is slower then native implementation. Library implementation can't generate native dynamic wrapper, it has to use slow libffi.

...

On Sat, 2014-03-15 at 00:17 -0400, Carter Schonwald wrote:

...
indeed, its very very easy to do storable instances that correspond to the struct type you want,

the ``with`` function in http://hackage.haskell.org/package/base-4.6.0.1/docs/Foreign-Marshal-Utils.h... actually gets you most of the way there!

Not sure I understand. `with` can be used to provide C function with pointer to C structure. How can it help when C function requires C structure to be passed by value?

...

i think the crux of Manuel's point is mainly that any good proposal has to at least give a roadmap to support on all the various platforms etc etc

...

From my point of view, at this point it is more important to agree on

I don't think you are expecting detailed schedule from me. Passing structure by value is possible on all platforms ghc supports, and it can be implemented for any particular platform if somebody is interested. the next question: do we want such functionality in ghc at all? I don't want to waste time on it if nobody wants to see it merged. Thanks, Yuras On Sat, 2014-03-15 at 00:37 -0400, Carter Schonwald wrote:

...

I'm not opposing that, in fact, theres a GHC ticket discussing some stuff related to this (related to complex numbers).

i think the crux of Manuel's point is mainly that any good proposal has to at least give a roadmap to support on all the various platforms etc etc

On Sat, Mar 15, 2014 at 12:33 AM, Edward Kmett wrote:

...
I don't care enough to fight and try to win the battle, but I just want to point out that Storable structs are far more brittle and platform dependent than borrowing the already correct platform logic for struct passing from libffi.

I do think the existing FFI extension made the right call under the 32 bit ABIs that were in use at the time it was defined. That said, with 64-bit ABIs saying that 2 32-bit ints should be passed in a single 64 bit register, you wind up with large chunks of third party APIs we just can't call out to directly any more, requiring many one-off manual C shims.

-Edward

On Sat, Mar 15, 2014 at 12:17 AM, Carter Schonwald < carter.schonwald@gmail.com> wrote:

...
indeed, its very very easy to do storable instances that correspond to the struct type you want,

the ``with`` function in http://hackage.haskell.org/package/base-4.6.0.1/docs/Foreign-Marshal-Utils.h... actually gets you most of the way there!

On Sat, Mar 15, 2014 at 12:00 AM, Manuel M T Chakravarty < chak@cse.unsw.edu.au> wrote:

...
Yuras,

I’m not convinced that the compiler is the right place for this kind of functionality. In fact, when we designed the Haskell FFI, we explicit decided against what you propose. There are a few reasons for this.

Firstly, compilers are complex beasts, and secondly, it takes a long time until a change in the compiler goes into production. Hence, as a general rule, it is advisable to move complexity from the compiler into libraries as this reduces compiler complexity. Libraries are less complex and changes can be rolled out much more quickly (it’s essentially a Hackage upload versus waiting for the next GHC and Haskell Platform release).

Thirdly, we have got the Haskell standard for a reason and modifying the compiler implies a language extension.

The design goal for the Haskell FFI was to provide the absolute minimum as part of the language and compiler, and to layer additional conveniences on top of that in the form of libraries and tools.

Have you considered the library or tool route?

Manuel

Yuras Shumovich :

...
Hi,

Right now ghc's FFI doesn't support c/c++ structures.

Whenever we have foreign function that accepts or returns struct by value, we have to create wrapper that accepts or returns pointer to struct. It is inconvenient, but actually not a big deal.

But there is no easy workaround when you want to export haskell function to use it with c/c++ API that requires structures to be passed by value (Usually it is a callback in c/c++ API. You can't change it's signature, and if it doesn't provide some kind of "void* userdata", then you are stuck.)

I'm interested in fixing that. I'm going to start with 'foreign import "wrapper" ...' stuff.

Calling conventions for passing c/c++ structures by value are pretty tricky and platform/compiler specific. So initially I'll use libffi for that (it will work when USE_LIBFFI_FOR_ADJUSTORS is defined, see rts/Adjustor.c). It will allow me to explore design space without bothering about low level implementation details. Later it could be implemented for native (non-libffi) adjustors.

Is anybody interested it that? I appreciate any comments/ideas.

Right now I don't have clear design. It would be nice to support plain haskell data types that are 1) not recursive, 2) has one constructor and 3) contains only c/c++ types. But it doesn't work with c/c++ unions. Any ideas are welcome.

An example how to use libffi with structures: http://www.atmark-techno.com/~yashi/libffi.html#Structures

Thanks, Yuras

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Manuel M T Chakravarty

17 Mar 17 Mar

9:37 p.m.

Yuras Shumovich :

...

I think the compiler is the right place. It is impossible to have efficient implementation in a library.

For dynamic wrapper (foreign import "wrapper" stuff) ghc generates piece of executable code at runtime. There are native implementations for a number of platforms, and libffi is used as a fall back for other platforms (see rts/Adjustor.c). AFAIK it is done that way because libffi is slower then native implementation.

Library implementation can't generate native dynamic wrapper, it has to use slow libffi.

When we first implemented the FFI, there was no libffi. Maintaining the adjustor code for all platforms is a PITA; hence, using libffi was a welcome way to improve portability. Making the adjustor code more complicated by adding more functionality doesn’t sound like a good plan to me. Besides, there are other overheads in addition to the actual marshalling in FFI calls and most of the time we are calling out to library functions for which the FFI call overhead is only a small portion of the runtime.

...

...
i think the crux of Manuel's point is mainly that any good proposal has to at least give a roadmap to support on all the various platforms etc etc

I don't think you are expecting detailed schedule from me. Passing structure by value is possible on all platforms ghc supports, and it can be implemented for any particular platform if somebody is interested.

From my point of view, at this point it is more important to agree on the next question: do we want such functionality in ghc at all? I don't want to waste time on it if nobody wants to see it merged.

I still don’t see the benefit in further complicating an already murky corner of the compiler. Moreover, for this to make sense, it would need to work on all supported platforms. Unless you are volunteering to implement it on multiple platforms, this would mean, we’d use libffi for most platforms anyway. This brings me to my original point, a library or tool is the better place for this. Manuel PS: I’d happily accept language-c-inline patches for marshalling structs.

...

On Sat, 2014-03-15 at 00:37 -0400, Carter Schonwald wrote:

...
I'm not opposing that, in fact, theres a GHC ticket discussing some stuff related to this (related to complex numbers).

i think the crux of Manuel's point is mainly that any good proposal has to at least give a roadmap to support on all the various platforms etc etc

On Sat, Mar 15, 2014 at 12:33 AM, Edward Kmett wrote:

...
I don't care enough to fight and try to win the battle, but I just want to point out that Storable structs are far more brittle and platform dependent than borrowing the already correct platform logic for struct passing from libffi.

I do think the existing FFI extension made the right call under the 32 bit ABIs that were in use at the time it was defined. That said, with 64-bit ABIs saying that 2 32-bit ints should be passed in a single 64 bit register, you wind up with large chunks of third party APIs we just can't call out to directly any more, requiring many one-off manual C shims.

-Edward

On Sat, Mar 15, 2014 at 12:17 AM, Carter Schonwald < carter.schonwald@gmail.com> wrote:

...
indeed, its very very easy to do storable instances that correspond to the struct type you want,

the ``with`` function in http://hackage.haskell.org/package/base-4.6.0.1/docs/Foreign-Marshal-Utils.h... actually gets you most of the way there!

On Sat, Mar 15, 2014 at 12:00 AM, Manuel M T Chakravarty < chak@cse.unsw.edu.au> wrote:

...
Yuras,

I’m not convinced that the compiler is the right place for this kind of functionality. In fact, when we designed the Haskell FFI, we explicit decided against what you propose. There are a few reasons for this.

Firstly, compilers are complex beasts, and secondly, it takes a long time until a change in the compiler goes into production. Hence, as a general rule, it is advisable to move complexity from the compiler into libraries as this reduces compiler complexity. Libraries are less complex and changes can be rolled out much more quickly (it’s essentially a Hackage upload versus waiting for the next GHC and Haskell Platform release).

Thirdly, we have got the Haskell standard for a reason and modifying the compiler implies a language extension.

The design goal for the Haskell FFI was to provide the absolute minimum as part of the language and compiler, and to layer additional conveniences on top of that in the form of libraries and tools.

Have you considered the library or tool route?

Manuel

Yuras Shumovich :

...
Hi,

Right now ghc's FFI doesn't support c/c++ structures.

Whenever we have foreign function that accepts or returns struct by value, we have to create wrapper that accepts or returns pointer to struct. It is inconvenient, but actually not a big deal.

But there is no easy workaround when you want to export haskell function to use it with c/c++ API that requires structures to be passed by value (Usually it is a callback in c/c++ API. You can't change it's signature, and if it doesn't provide some kind of "void* userdata", then you are stuck.)

I'm interested in fixing that. I'm going to start with 'foreign import "wrapper" ...' stuff.

Calling conventions for passing c/c++ structures by value are pretty tricky and platform/compiler specific. So initially I'll use libffi for that (it will work when USE_LIBFFI_FOR_ADJUSTORS is defined, see rts/Adjustor.c). It will allow me to explore design space without bothering about low level implementation details. Later it could be implemented for native (non-libffi) adjustors.

Is anybody interested it that? I appreciate any comments/ideas.

Right now I don't have clear design. It would be nice to support plain haskell data types that are 1) not recursive, 2) has one constructor and 3) contains only c/c++ types. But it doesn't work with c/c++ unions. Any ideas are welcome.

An example how to use libffi with structures: http://www.atmark-techno.com/~yashi/libffi.html#Structures

Thanks, Yuras

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Yuras Shumovich

18 Mar 18 Mar

7:42 a.m.

On Tue, 2014-03-18 at 12:37 +1100, Manuel M T Chakravarty wrote:

...

...
Library implementation can't generate native dynamic wrapper, it has to use slow libffi.

When we first implemented the FFI, there was no libffi. Maintaining the adjustor code for all platforms is a PITA; hence, using libffi was a welcome way to improve portability.

Do you think we can remove native adjustors? I can prepare a patch. It requires minor changes to cache ffi_cif structure. On desugar phase for each wrapper we can generate fresh global variable to store cif pointer and pass it to createAdjustor.

...

...
From my point of view, at this point it is more important to agree on the next question: do we want such functionality in ghc at all? I don't want to waste time on it if nobody wants to see it merged.

I still don’t see the benefit in further complicating an already murky corner of the compiler. Moreover, for this to make sense, it would need to work on all supported platforms. Unless you are volunteering to implement it on multiple platforms, this would mean, we’d use libffi for most platforms anyway. This brings me to my original point, a library or tool is the better place for this.

OK, I don't buy it, but I see your point.

...

Manuel

PS: I’d happily accept language-c-inline patches for marshalling structs.

Simon Marlow

19 Mar 19 Mar

8:12 a.m.

On 18/03/2014 11:42, Yuras Shumovich wrote:

...

On Tue, 2014-03-18 at 12:37 +1100, Manuel M T Chakravarty wrote:

...
...
Library implementation can't generate native dynamic wrapper, it has to use slow libffi.

When we first implemented the FFI, there was no libffi. Maintaining the adjustor code for all platforms is a PITA; hence, using libffi was a welcome way to improve portability.

Do you think we can remove native adjustors? I can prepare a patch.

I believe native adjustors are sufficiently faster than libffi to want to keep them; at least that was what I found the last time I checked. It might be the case that either libffi is now faster, or we could optimise our libffi support enough that it isn't worth keeping native adjustors. I'd love it if that happened. The benchmark I think I used is in nofib/smp/callback002. Cheers, Simon

...

It requires minor changes to cache ffi_cif structure. On desugar phase for each wrapper we can generate fresh global variable to store cif pointer and pass it to createAdjustor.

...
...
From my point of view, at this point it is more important to agree on the next question: do we want such functionality in ghc at all? I don't want to waste time on it if nobody wants to see it merged.

I still don’t see the benefit in further complicating an already murky corner of the compiler. Moreover, for this to make sense, it would need to work on all supported platforms. Unless you are volunteering to implement it on multiple platforms, this would mean, we’d use libffi for most platforms anyway. This brings me to my original point, a library or tool is the better place for this.

OK, I don't buy it, but I see your point.

...
Manuel

PS: I’d happily accept language-c-inline patches for marshalling structs.

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Yuras Shumovich

20 Mar 20 Mar

10:38 a.m.

On Wed, 2014-03-19 at 12:12 +0000, Simon Marlow wrote:

...

On 18/03/2014 11:42, Yuras Shumovich wrote:

...
On Tue, 2014-03-18 at 12:37 +1100, Manuel M T Chakravarty wrote:

...
...
Library implementation can't generate native dynamic wrapper, it has to use slow libffi.

When we first implemented the FFI, there was no libffi. Maintaining the adjustor code for all platforms is a PITA; hence, using libffi was a welcome way to improve portability.

Do you think we can remove native adjustors? I can prepare a patch.

I believe native adjustors are sufficiently faster than libffi to want to keep them; at least that was what I found the last time I checked. It might be the case that either libffi is now faster, or we could optimise our libffi support enough that it isn't worth keeping native adjustors. I'd love it if that happened.

The benchmark I think I used is in nofib/smp/callback002.

callback002 is 10-15% faster with native adjustors then with libffi (x86_64 linux, BuildFlavour = perf, mode=slow). Not a huge difference, but it is noticeable. Native adjustors: 6.53user 0.15system 0:06.69elapsed 99%CPU (0avgtext+0avgdata 2436maxresident)k 0inputs+8outputs (0major+894minor)pagefaults 0swaps <> With "UseLibFFIForAdjustors=YES" in mk/build.mk: 7.18user 0.10system 0:07.29elapsed 99%CPU (0avgtext+0avgdata 2440maxresident)k 0inputs+8outputs (0major+893minor)pagefaults 0swaps <> However, as Sergei Trofimovich mentioned, we use native adjustors only on i386 and x86_64. So rts/AdjustorAsm.S and half of rts/Adjustor.c seems to be a dead code. Unless someone has "UseLibFFIForAdjustors=NO" in his mk/build.mk. Thanks, Yuras

...

Cheers, Simon

...
It requires minor changes to cache ffi_cif structure. On desugar phase for each wrapper we can generate fresh global variable to store cif pointer and pass it to createAdjustor.

...
...
From my point of view, at this point it is more important to agree on the next question: do we want such functionality in ghc at all? I don't want to waste time on it if nobody wants to see it merged.

I still don’t see the benefit in further complicating an already murky corner of the compiler. Moreover, for this to make sense, it would need to work on all supported platforms. Unless you are volunteering to implement it on multiple platforms, this would mean, we’d use libffi for most platforms anyway. This brings me to my original point, a library or tool is the better place for this.

OK, I don't buy it, but I see your point.

...
Manuel

PS: I’d happily accept language-c-inline patches for marshalling structs.

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Simon Marlow

18 Mar 18 Mar

8:19 a.m.

On 18/03/2014 01:37, Manuel M T Chakravarty wrote:

...

Yuras Shumovich :

...
I think the compiler is the right place. It is impossible to have efficient implementation in a library.

For dynamic wrapper (foreign import "wrapper" stuff) ghc generates piece of executable code at runtime. There are native implementations for a number of platforms, and libffi is used as a fall back for other platforms (see rts/Adjustor.c). AFAIK it is done that way because libffi is slower then native implementation.

Library implementation can't generate native dynamic wrapper, it has to use slow libffi.

When we first implemented the FFI, there was no libffi. Maintaining the adjustor code for all platforms is a PITA; hence, using libffi was a welcome way to improve portability.

Making the adjustor code more complicated by adding more functionality doesn’t sound like a good plan to me.

Besides, there are other overheads in addition to the actual marshalling in FFI calls and most of the time we are calling out to library functions for which the FFI call overhead is only a small portion of the runtime.

...
...
i think the crux of Manuel's point is mainly that any good proposal has to at least give a roadmap to support on all the various platforms etc etc

I don't think you are expecting detailed schedule from me. Passing structure by value is possible on all platforms ghc supports, and it can be implemented for any particular platform if somebody is interested.

From my point of view, at this point it is more important to agree on the next question: do we want such functionality in ghc at all? I don't want to waste time on it if nobody wants to see it merged.

I'm really keen to have support for returning structs in particular. Passing structs less so, because working around the lack of struct passing isn't nearly as onerous as working around the lack of struct returns. Returning multiple values from a C function is a real pain without struct returns: you have to either allocate some memory in Haskell or in C, and both methods are needlessly complex and slow. (though allocating in Haskell is usually better.) C++ code does this all the time, so if you're wrapping C++ code for calling from Haskell, the lack of multiple returns bites a lot. In fact implementing this is on my todo list, I'm really glad to see someone else is planning to do it :-) The vague plan I had in my head was to allow the return value of a foreign import to be a tuple containing marshallable types, which would map to the appropriate return convention for a struct on the current platform. Perhaps allowing it to be an arbitrary single-constructor type is better, because it allows us to use a type that has a Storable instance. Cheers, Simon

...

I still don’t see the benefit in further complicating an already murky corner of the compiler. Moreover, for this to make sense, it would need to work on all supported platforms. Unless you are volunteering to implement it on multiple platforms, this would mean, we’d use libffi for most platforms anyway. This brings me to my original point, a library or tool is the better place for this.

Manuel

PS: I’d happily accept language-c-inline patches for marshalling structs.

...
On Sat, 2014-03-15 at 00:37 -0400, Carter Schonwald wrote:

...
I'm not opposing that, in fact, theres a GHC ticket discussing some stuff related to this (related to complex numbers).

i think the crux of Manuel's point is mainly that any good proposal has to at least give a roadmap to support on all the various platforms etc etc

On Sat, Mar 15, 2014 at 12:33 AM, Edward Kmett wrote:

...
I don't care enough to fight and try to win the battle, but I just want to point out that Storable structs are far more brittle and platform dependent than borrowing the already correct platform logic for struct passing from libffi.

I do think the existing FFI extension made the right call under the 32 bit ABIs that were in use at the time it was defined. That said, with 64-bit ABIs saying that 2 32-bit ints should be passed in a single 64 bit register, you wind up with large chunks of third party APIs we just can't call out to directly any more, requiring many one-off manual C shims.

-Edward

On Sat, Mar 15, 2014 at 12:17 AM, Carter Schonwald < carter.schonwald@gmail.com> wrote:

...
indeed, its very very easy to do storable instances that correspond to the struct type you want,

the ``with`` function in http://hackage.haskell.org/package/base-4.6.0.1/docs/Foreign-Marshal-Utils.h... actually gets you most of the way there!

On Sat, Mar 15, 2014 at 12:00 AM, Manuel M T Chakravarty < chak@cse.unsw.edu.au> wrote:

...
Yuras,

I’m not convinced that the compiler is the right place for this kind of functionality. In fact, when we designed the Haskell FFI, we explicit decided against what you propose. There are a few reasons for this.

Firstly, compilers are complex beasts, and secondly, it takes a long time until a change in the compiler goes into production. Hence, as a general rule, it is advisable to move complexity from the compiler into libraries as this reduces compiler complexity. Libraries are less complex and changes can be rolled out much more quickly (it’s essentially a Hackage upload versus waiting for the next GHC and Haskell Platform release).

Thirdly, we have got the Haskell standard for a reason and modifying the compiler implies a language extension.

The design goal for the Haskell FFI was to provide the absolute minimum as part of the language and compiler, and to layer additional conveniences on top of that in the form of libraries and tools.

Have you considered the library or tool route?

Manuel

Yuras Shumovich : > Hi, > > Right now ghc's FFI doesn't support c/c++ structures. > > Whenever we have foreign function that accepts or returns struct by > value, we have to create wrapper that accepts or returns pointer to > struct. It is inconvenient, but actually not a big deal. > > But there is no easy workaround when you want to export haskell function > to use it with c/c++ API that requires structures to be passed by value > (Usually it is a callback in c/c++ API. You can't change it's signature, > and if it doesn't provide some kind of "void* userdata", then you are > stuck.) > > I'm interested in fixing that. I'm going to start with 'foreign import > "wrapper" ...' stuff. > > Calling conventions for passing c/c++ structures by value are pretty > tricky and platform/compiler specific. So initially I'll use libffi for > that (it will work when USE_LIBFFI_FOR_ADJUSTORS is defined, see > rts/Adjustor.c). It will allow me to explore design space without > bothering about low level implementation details. Later it could be > implemented for native (non-libffi) adjustors. > > Is anybody interested it that? I appreciate any comments/ideas. > > Right now I don't have clear design. It would be nice to support plain > haskell data types that are 1) not recursive, 2) has one constructor and > 3) contains only c/c++ types. But it doesn't work with c/c++ unions. Any > ideas are welcome. > > An example how to use libffi with structures: > http://www.atmark-techno.com/~yashi/libffi.html#Structures > > Thanks, > Yuras > > > _______________________________________________ > ghc-devs mailing list > ghc-devs@haskell.org > http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Yuras Shumovich

1:31 p.m.

Hi, I thought I have lost the battle :) Thank you for the support, Simon! I'm interested in full featured solution: arguments, return value, foreign import, foreign export, etc. But it is too much for me to do it all at once. So I started with dynamic wrapper. The plan is to support structs as arguments and return value for dynamic wrapper using libffi; then implement native adjustors at least for x86_64 linux; then make final design decision (tuple or data? language pragma? union support? etc); and only then start working on foreign import. But I'm open for suggestions. Just let me know if you think it is better to start with return value support for foreign import. Thanks, Yuras On Tue, 2014-03-18 at 12:19 +0000, Simon Marlow wrote:

...

I'm really keen to have support for returning structs in particular. Passing structs less so, because working around the lack of struct passing isn't nearly as onerous as working around the lack of struct returns. Returning multiple values from a C function is a real pain without struct returns: you have to either allocate some memory in Haskell or in C, and both methods are needlessly complex and slow. (though allocating in Haskell is usually better.) C++ code does this all the time, so if you're wrapping C++ code for calling from Haskell, the lack of multiple returns bites a lot.

In fact implementing this is on my todo list, I'm really glad to see someone else is planning to do it :-)

The vague plan I had in my head was to allow the return value of a foreign import to be a tuple containing marshallable types, which would map to the appropriate return convention for a struct on the current platform. Perhaps allowing it to be an arbitrary single-constructor type is better, because it allows us to use a type that has a Storable instance.

Cheers, Simon

Simon Marlow

5:30 p.m.

So the hard parts are: - the native code generators - native adjustor support (rts/Adjustor.c) Everything else is relatively striaghtforward: we use libffi for adjustors on some platforms and for GHCi, and the LLVM backend should be quite easy too. I would at least take a look at the hard bits and see whether you think it's going to be possible to extend these to handle struct args/returns. Because if not, then the idea is a dead end. Or maybe we will need to limit the scope to make things easier (e.g. only integer and pointer fields). Cheers, Simon On 18/03/2014 17:31, Yuras Shumovich wrote:

...

Hi,

I thought I have lost the battle :) Thank you for the support, Simon!

I'm interested in full featured solution: arguments, return value, foreign import, foreign export, etc. But it is too much for me to do it all at once. So I started with dynamic wrapper.

The plan is to support structs as arguments and return value for dynamic wrapper using libffi; then implement native adjustors at least for x86_64 linux; then make final design decision (tuple or data? language pragma? union support? etc); and only then start working on foreign import.

But I'm open for suggestions. Just let me know if you think it is better to start with return value support for foreign import.

Thanks, Yuras

On Tue, 2014-03-18 at 12:19 +0000, Simon Marlow wrote:

...
I'm really keen to have support for returning structs in particular. Passing structs less so, because working around the lack of struct passing isn't nearly as onerous as working around the lack of struct returns. Returning multiple values from a C function is a real pain without struct returns: you have to either allocate some memory in Haskell or in C, and both methods are needlessly complex and slow. (though allocating in Haskell is usually better.) C++ code does this all the time, so if you're wrapping C++ code for calling from Haskell, the lack of multiple returns bites a lot.

In fact implementing this is on my todo list, I'm really glad to see someone else is planning to do it :-)

The vague plan I had in my head was to allow the return value of a foreign import to be a tuple containing marshallable types, which would map to the appropriate return convention for a struct on the current platform. Perhaps allowing it to be an arbitrary single-constructor type is better, because it allows us to use a type that has a Storable instance.

Cheers, Simon

Yuras Shumovich

14 Jun 14 Jun

11:57 a.m.

Hello, I implemented support for returning C structures by value in cmm for x86_64 (most likely it works only on linux). You can find it here: https://github.com/Yuras/ghc/commits/cmm-cstruct It supports I8, I16, I32, I64, F_, D_ cmm types, and requires special annotation. For example: #include "Cmm.h" #define MyStruct struct(CInt, I8, struct(I8, CInt)) cmm_test(W_ i) { CInt i1; I8 i2, i3; float32 i4; (i1, i2, i3, i4) = ccall c_test(W_TO_INT(i)) MyStruct; return (TO_W_(i1), TO_W_(i2), TO_W_(i3), i4); } (See "test" directory for full examples.) Do you think it is right approach? Could anyone review the code please? And the last thing, I need mentor for this project. Is anyone interested? Thanks, Yuras On Tue, 2014-03-18 at 21:30 +0000, Simon Marlow wrote:

...

So the hard parts are:

- the native code generators - native adjustor support (rts/Adjustor.c)

Everything else is relatively striaghtforward: we use libffi for adjustors on some platforms and for GHCi, and the LLVM backend should be quite easy too.

I would at least take a look at the hard bits and see whether you think it's going to be possible to extend these to handle struct args/returns. Because if not, then the idea is a dead end. Or maybe we will need to limit the scope to make things easier (e.g. only integer and pointer fields).

Cheers, Simon

On 18/03/2014 17:31, Yuras Shumovich wrote:

...
Hi,

I thought I have lost the battle :) Thank you for the support, Simon!

I'm interested in full featured solution: arguments, return value, foreign import, foreign export, etc. But it is too much for me to do it all at once. So I started with dynamic wrapper.

The plan is to support structs as arguments and return value for dynamic wrapper using libffi; then implement native adjustors at least for x86_64 linux; then make final design decision (tuple or data? language pragma? union support? etc); and only then start working on foreign import.

But I'm open for suggestions. Just let me know if you think it is better to start with return value support for foreign import.

Thanks, Yuras

On Tue, 2014-03-18 at 12:19 +0000, Simon Marlow wrote:

...
I'm really keen to have support for returning structs in particular. Passing structs less so, because working around the lack of struct passing isn't nearly as onerous as working around the lack of struct returns. Returning multiple values from a C function is a real pain without struct returns: you have to either allocate some memory in Haskell or in C, and both methods are needlessly complex and slow. (though allocating in Haskell is usually better.) C++ code does this all the time, so if you're wrapping C++ code for calling from Haskell, the lack of multiple returns bites a lot.

In fact implementing this is on my todo list, I'm really glad to see someone else is planning to do it :-)

The vague plan I had in my head was to allow the return value of a foreign import to be a tuple containing marshallable types, which would map to the appropriate return convention for a struct on the current platform. Perhaps allowing it to be an arbitrary single-constructor type is better, because it allows us to use a type that has a Storable instance.

Cheers, Simon

Iavor Diatchki

9:04 p.m.

Hello, just as another data point: I'd also find this functionality useful. I've run into the problem when trying to bind to C libraries that have functions with struct parameters being passed by value, or returned (the idea being that under the hood these structs often end up split up across registers). In the library I was working with, colors were implemented as RGBA structs which were passed and returned by value all over the place. -Iavor On Sat, Jun 14, 2014 at 8:57 AM, Yuras Shumovich wrote:

...

Hello,

I implemented support for returning C structures by value in cmm for x86_64 (most likely it works only on linux). You can find it here: https://github.com/Yuras/ghc/commits/cmm-cstruct

It supports I8, I16, I32, I64, F_, D_ cmm types, and requires special annotation. For example:

#include "Cmm.h"

#define MyStruct struct(CInt, I8, struct(I8, CInt))

cmm_test(W_ i) { CInt i1; I8 i2, i3; float32 i4; (i1, i2, i3, i4) = ccall c_test(W_TO_INT(i)) MyStruct; return (TO_W_(i1), TO_W_(i2), TO_W_(i3), i4); }

(See "test" directory for full examples.)

Do you think it is right approach? Could anyone review the code please?

And the last thing, I need mentor for this project. Is anyone interested?

Thanks, Yuras

On Tue, 2014-03-18 at 21:30 +0000, Simon Marlow wrote:

...
So the hard parts are:

- the native code generators - native adjustor support (rts/Adjustor.c)

Everything else is relatively striaghtforward: we use libffi for adjustors on some platforms and for GHCi, and the LLVM backend should be quite easy too.

I would at least take a look at the hard bits and see whether you think it's going to be possible to extend these to handle struct args/returns. Because if not, then the idea is a dead end. Or maybe we will need to limit the scope to make things easier (e.g. only integer and pointer fields).

Cheers, Simon

On 18/03/2014 17:31, Yuras Shumovich wrote:

...
Hi,

I thought I have lost the battle :) Thank you for the support, Simon!

I'm interested in full featured solution: arguments, return value, foreign import, foreign export, etc. But it is too much for me to do it all at once. So I started with dynamic wrapper.

The plan is to support structs as arguments and return value for dynamic wrapper using libffi; then implement native adjustors at least for x86_64 linux; then make final design decision (tuple or data? language pragma? union support? etc); and only then start working on foreign import.

But I'm open for suggestions. Just let me know if you think it is better to start with return value support for foreign import.

Thanks, Yuras

On Tue, 2014-03-18 at 12:19 +0000, Simon Marlow wrote:

...
I'm really keen to have support for returning structs in particular. Passing structs less so, because working around the lack of struct passing isn't nearly as onerous as working around the lack of struct returns. Returning multiple values from a C function is a real pain without struct returns: you have to either allocate some memory in Haskell or in C, and both methods are needlessly complex and slow. (though allocating in Haskell is usually better.) C++ code does this all the time, so if you're wrapping C++ code for calling from Haskell, the lack of multiple returns bites a lot.

In fact implementing this is on my todo list, I'm really glad to see someone else is planning to do it :-)

The vague plan I had in my head was to allow the return value of a foreign import to be a tuple containing marshallable types, which would map to the appropriate return convention for a struct on the current platform. Perhaps allowing it to be an arbitrary single-constructor type is better, because it allows us to use a type that has a Storable instance.

Cheers, Simon

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Yuras Shumovich

7 Oct 7 Oct

12:59 p.m.

Simon, I finally managed to implement that for major NCG backends. Phabricator revision is here: https://phabricator.haskell.org/D252 Here is a link to the review you did before: https://github.com/Yuras/ghc/commit/7295a4c600bc69129b6800be5b52c3842c9c4e5b I don't have implementation for mac os x86, ppc and sparc. Are they actively used today? I don't have access to hardware to test them. Do you think it has it's own value without exposing to haskell FFI? What is the minimal feature set I should implement to make it merged? Thanks, Yuras On Sat, 2014-06-14 at 18:57 +0300, Yuras Shumovich wrote:

...

Hello,

I implemented support for returning C structures by value in cmm for x86_64 (most likely it works only on linux). You can find it here: https://github.com/Yuras/ghc/commits/cmm-cstruct

It supports I8, I16, I32, I64, F_, D_ cmm types, and requires special annotation. For example:

#include "Cmm.h"

#define MyStruct struct(CInt, I8, struct(I8, CInt))

cmm_test(W_ i) { CInt i1; I8 i2, i3; float32 i4; (i1, i2, i3, i4) = ccall c_test(W_TO_INT(i)) MyStruct; return (TO_W_(i1), TO_W_(i2), TO_W_(i3), i4); }

(See "test" directory for full examples.)

Do you think it is right approach? Could anyone review the code please?

And the last thing, I need mentor for this project. Is anyone interested?

Thanks, Yuras

On Tue, 2014-03-18 at 21:30 +0000, Simon Marlow wrote:

...
So the hard parts are:

- the native code generators - native adjustor support (rts/Adjustor.c)

Everything else is relatively striaghtforward: we use libffi for adjustors on some platforms and for GHCi, and the LLVM backend should be quite easy too.

I would at least take a look at the hard bits and see whether you think it's going to be possible to extend these to handle struct args/returns. Because if not, then the idea is a dead end. Or maybe we will need to limit the scope to make things easier (e.g. only integer and pointer fields).

Cheers, Simon

On 18/03/2014 17:31, Yuras Shumovich wrote:

...
Hi,

I thought I have lost the battle :) Thank you for the support, Simon!

I'm interested in full featured solution: arguments, return value, foreign import, foreign export, etc. But it is too much for me to do it all at once. So I started with dynamic wrapper.

The plan is to support structs as arguments and return value for dynamic wrapper using libffi; then implement native adjustors at least for x86_64 linux; then make final design decision (tuple or data? language pragma? union support? etc); and only then start working on foreign import.

But I'm open for suggestions. Just let me know if you think it is better to start with return value support for foreign import.

Thanks, Yuras

On Tue, 2014-03-18 at 12:19 +0000, Simon Marlow wrote:

...
I'm really keen to have support for returning structs in particular. Passing structs less so, because working around the lack of struct passing isn't nearly as onerous as working around the lack of struct returns. Returning multiple values from a C function is a real pain without struct returns: you have to either allocate some memory in Haskell or in C, and both methods are needlessly complex and slow. (though allocating in Haskell is usually better.) C++ code does this all the time, so if you're wrapping C++ code for calling from Haskell, the lack of multiple returns bites a lot.

In fact implementing this is on my todo list, I'm really glad to see someone else is planning to do it :-)

The vague plan I had in my head was to allow the return value of a foreign import to be a tuple containing marshallable types, which would map to the appropriate return convention for a struct on the current platform. Perhaps allowing it to be an arbitrary single-constructor type is better, because it allows us to use a type that has a Storable instance.

Cheers, Simon

Sergei Trofimovich

19 Mar 19 Mar

3:45 a.m.

On Tue, 18 Mar 2014 20:31:19 +0300 Yuras Shumovich wrote:

...

Hi,

I thought I have lost the battle :) Thank you for the support, Simon!

I'm interested in full featured solution: arguments, return value, foreign import, foreign export, etc. But it is too much for me to do it all at once. So I started with dynamic wrapper.

The plan is to support structs as arguments and return value for dynamic wrapper using libffi; then implement native adjustors at least for x86_64 linux;

On a positive side there is only 2 arches supporting native adjustors that actually work these days: ArchHasAdjustorSupport = $(if $(findstring $(TargetArch_CPP),i386 x86_64),YES,NO)

...

then make final design decision (tuple or data? language pragma? union support? etc); and only then start working on foreign import.

But I'm open for suggestions. Just let me know if you think it is better to start with return value support for foreign import.

Thanks, Yuras

-- Sergei

3878

Age (days ago)

4085

Last active (days ago)

List overview

Download

19 comments

7 participants

participants (7)

Carter Schonwald
Edward Kmett
Iavor Diatchki
Manuel M T Chakravarty
Sergei Trofimovich
Simon Marlow
Yuras Shumovich