ANNOUNCE: cinvoke 0.1 released

newer
Why don't my OS threads terminate?

older
ANNOUNCE: Haskell Platform 2011.2...

Remi Turk

6 Mar 2011 6 Mar '11

8:38 a.m.

I am happy to finally announce cinvoke 0.1, a binding to the C library cinvoke[1], allowing functions to be loaded and called whose names and types are not known before run-time. Why? Sometimes you can't use the Haskell foreign function interface because you parse the type of the function from somewhere else, i.e. you're writing an interpreter for a language that has an FFI itself. What? The main function it exports is: cinvoke :: Symbol -> RetType b -> [Arg] -> IO b And because code is worth a thousand words, here's a small program that uses libc to write a 1Gb buffer of random garbage to a file:

...

module Main where

import Foreign.CInvoke

main = do cxt <- newContext libc <- loadLibrary cxt "libc.so.6" malloc <- loadSymbol libc "malloc" creat <- loadSymbol libc "creat" write <- loadSymbol libc "write" free <- loadSymbol libc "free" let sz = 2^30 buf <- cinvoke malloc (retPtr retVoid) [argCSize sz] fd <- cinvoke creat retCInt [argString "/tmp/test", argCUInt 0o644] n <- cinvoke write retCSize [argCInt fd, argPtr buf, argCSize sz] cinvoke free (retPtr retVoid) [argPtr buf]

It hopefully works on any machine on which cinvoke works, but has only been tested on linux x86_64. As the current version of cinvoke only installs a static library, it does not work from GHCi at the moment (without hacking cinvoke to build a shared library). More interesting examples are included in examples/ in the package. Where? Hackage: http://hackage.haskell.org/package/cinvoke Cheers, Remi [1] http://www.nongnu.org/cinvoke/

Show replies by date

Yves Parès

6 Mar 6 Mar

6:27 p.m.

It evokes me Python's ctypes module. Nice job ! I think It will be useful. 2011/3/6 Remi Turk

...

I am happy to finally announce cinvoke 0.1, a binding to the C library cinvoke[1], allowing functions to be loaded and called whose names and types are not known before run-time.

Why?

Sometimes you can't use the Haskell foreign function interface because you parse the type of the function from somewhere else, i.e. you're writing an interpreter for a language that has an FFI itself.

What?

The main function it exports is:

cinvoke :: Symbol -> RetType b -> [Arg] -> IO b

And because code is worth a thousand words, here's a small program that uses libc to write a 1Gb buffer of random garbage to a file:

...
module Main where

import Foreign.CInvoke

main = do cxt <- newContext libc <- loadLibrary cxt "libc.so.6" malloc <- loadSymbol libc "malloc" creat <- loadSymbol libc "creat" write <- loadSymbol libc "write" free <- loadSymbol libc "free" let sz = 2^30 buf <- cinvoke malloc (retPtr retVoid) [argCSize sz] fd <- cinvoke creat retCInt [argString "/tmp/test", argCUInt 0o644] n <- cinvoke write retCSize [argCInt fd, argPtr buf, argCSize sz] cinvoke free (retPtr retVoid) [argPtr buf]

It hopefully works on any machine on which cinvoke works, but has only been tested on linux x86_64. As the current version of cinvoke only installs a static library, it does not work from GHCi at the moment (without hacking cinvoke to build a shared library). More interesting examples are included in examples/ in the package.

Where? Hackage: http://hackage.haskell.org/package/cinvoke

Cheers, Remi

[1] http://www.nongnu.org/cinvoke/

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Max Bolingbroke

7 Mar 7 Mar

4:41 a.m.

Hi Remi, On 6 March 2011 13:38, Remi Turk wrote:

...

I am happy to finally announce cinvoke 0.1, a binding to the C library cinvoke[1], allowing functions to be loaded and called whose names and types are not known before run-time.

As the author of the libffi package (http://hackage.haskell.org/package/libffi-0.1) which does a similar thing, could you say when it would be appropriate to use one or the other package? Cheers, Max

Remi Turk

1:32 p.m.

On Mon, Mar 07, 2011 at 09:41:27AM +0000, Max Bolingbroke wrote:

...

Hi Remi,

On 6 March 2011 13:38, Remi Turk wrote:

...
I am happy to finally announce cinvoke 0.1, a binding to the C library cinvoke[1], allowing functions to be loaded and called whose names and types are not known before run-time.

As the author of the libffi package (http://hackage.haskell.org/package/libffi-0.1) which does a similar thing, could you say when it would be appropriate to use one or the other package?

Cheers, Max

Of course: - libffi doesn't do library/function loading; you'll need to use System.Posix.DynamicLinker or System.Win32.DLL for that. cinvoke will not only load your libraries and functions, but even collect the garbage afterwards. - Things seem to have changed, but back when I first looked at cinvoke, getting libffi to run under windows didn't seem too realistic. - If you need to pass C structs (by value), you'll have to use libffi: cinvoke doesn't support them at all. - The current version of libffi is not exception safe (I do have some code lying around here though...) - cinvoke is actually haddockized (although hackage still hasn't generated the docs, apparently). Groeten, Remi

Gábor Lehel

3:42 p.m.

On Mon, Mar 7, 2011 at 7:32 PM, Remi Turk wrote:

...

On Mon, Mar 07, 2011 at 09:41:27AM +0000, Max Bolingbroke wrote:

...
Hi Remi,

On 6 March 2011 13:38, Remi Turk wrote:

...
I am happy to finally announce cinvoke 0.1, a binding to the C library cinvoke[1], allowing functions to be loaded and called whose names and types are not known before run-time.

As the author of the libffi package (http://hackage.haskell.org/package/libffi-0.1) which does a similar thing, could you say when it would be appropriate to use one or the other package?

Cheers, Max

Of course:

- libffi doesn't do library/function loading; you'll need to use System.Posix.DynamicLinker or System.Win32.DLL for that. cinvoke will not only load your libraries and functions, but even collect the garbage afterwards. - Things seem to have changed, but back when I first looked at cinvoke, getting libffi to run under windows didn't seem too realistic. - If you need to pass C structs (by value), you'll have to use libffi: cinvoke doesn't support them at all. - The current version of libffi is not exception safe (I do have some code lying around here though...) - cinvoke is actually haddockized (although hackage still hasn't generated the docs, apparently).

It's reporting a build failure.

...

Groeten, Remi

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- Work is punishment for failing to procrastinate effectively.

Daniel Fischer

4 p.m.

On Monday 07 March 2011 21:42:16, Gábor Lehel wrote:

...

It's reporting a build failure.

Missing C library.

Remi Turk

4:14 p.m.

On Mon, Mar 07, 2011 at 10:00:47PM +0100, Daniel Fischer wrote:

...

On Monday 07 March 2011 21:42:16, Gábor Lehel wrote:

...
It's reporting a build failure.

Missing C library.

cinvoke (the C library) is obviously not installed on the testing machine. Does that really mean no library with uncommon C dependencies gets documentation on hackage? Remi

Daniel Fischer

4:31 p.m.

On Monday 07 March 2011 22:14:38, Remi Turk wrote:

...

On Mon, Mar 07, 2011 at 10:00:47PM +0100, Daniel Fischer wrote:

...
On Monday 07 March 2011 21:42:16, Gábor Lehel wrote:

...
It's reporting a build failure.

Missing C library.

cinvoke (the C library) is obviously not installed on the testing machine. Does that really mean no library with uncommon C dependencies gets documentation on hackage?

Remi

Basically, yes. As far as I know, documentation is only built for libraries that build on hackage. Maybe it would be a good idea to have the opportunity to upload haddock bundles to hackage too for such libraries.

Remi Turk

4:58 p.m.

On Mon, Mar 07, 2011 at 10:31:25PM +0100, Daniel Fischer wrote:

...

On Monday 07 March 2011 22:14:38, Remi Turk wrote:

...
cinvoke (the C library) is obviously not installed on the testing machine. Does that really mean no library with uncommon C dependencies gets documentation on hackage?

Remi

Basically, yes. As far as I know, documentation is only built for libraries that build on hackage.

Maybe it would be a good idea to have the opportunity to upload haddock bundles to hackage too for such libraries.

That sucks :( Uploading haddock bundles could solve the problem, though I don't currently understand why being able to successfully configure a package is a prerequisite to generating the docs. Anyway, I just put the docs online somewhere else with a link from the homepage: http://haskell.org/haskellwiki/Library/cinvoke

Felipe Almeida Lessa

8:15 p.m.

On Mon, Mar 7, 2011 at 6:32 PM, Remi Turk wrote:

...

- If you need to pass C structs (by value), you'll have to use libffi: cinvoke doesn't support them at all.

What about CInvStructure[1]? I was just glancing at the documentation when I saw this. Cheers! =) [1] http://www.nongnu.org/cinvoke/doc/cinvoke_8h.html#b39daa4325b8b87aa246cd8acf... -- Felipe.

Remi Turk

8 Mar 8 Mar

5:36 a.m.

On Tue, Mar 08, 2011 at 01:15:26AM +0000, Felipe Almeida Lessa wrote:

...

On Mon, Mar 7, 2011 at 6:32 PM, Remi Turk wrote:

...
- If you need to pass C structs (by value), you'll have to use libffi: cinvoke doesn't support them at all.

What about CInvStructure[1]? I was just glancing at the documentation when I saw this.

That's a part of cinvoke I have not implemented (and probably won't, just like callbacks and a few other things, at least until there is some demand for them). However, the CInvStructure functions are used to construct descriptions and instances of C structures at run-time. (think alignment issues...) Passing structures to functions using cinvoke can only be done using pointers though.[1] Cheers, Remi [1] http://www.nongnu.org/cinvoke/doc/cinvoke_8h.html#4d288cacc9bde484cad7d8ed1b...

Gábor Lehel

7:01 a.m.

On Sun, Mar 6, 2011 at 2:38 PM, Remi Turk wrote:

...

I am happy to finally announce cinvoke 0.1, a binding to the C library cinvoke[1], allowing functions to be loaded and called whose names and types are not known before run-time.

Why?

Sometimes you can't use the Haskell foreign function interface because you parse the type of the function from somewhere else, i.e. you're writing an interpreter for a language that has an FFI itself.

What?

The main function it exports is:

cinvoke :: Symbol -> RetType b -> [Arg] -> IO b

And because code is worth a thousand words, here's a small program that uses libc to write a 1Gb buffer of random garbage to a file:

...
module Main where

import Foreign.CInvoke

main = do cxt <- newContext libc <- loadLibrary cxt "libc.so.6" malloc <- loadSymbol libc "malloc" creat <- loadSymbol libc "creat" write <- loadSymbol libc "write" free <- loadSymbol libc "free" let sz = 2^30 buf <- cinvoke malloc (retPtr retVoid) [argCSize sz] fd <- cinvoke creat retCInt [argString "/tmp/test", argCUInt 0o644] n <- cinvoke write retCSize [argCInt fd, argPtr buf, argCSize sz] cinvoke free (retPtr retVoid) [argPtr buf]

It hopefully works on any machine on which cinvoke works, but has only been tested on linux x86_64. As the current version of cinvoke only installs a static library, it does not work from GHCi at the moment (without hacking cinvoke to build a shared library). More interesting examples are included in examples/ in the package.

Where? Hackage: http://hackage.haskell.org/package/cinvoke

Cheers, Remi

[1] http://www.nongnu.org/cinvoke/

Is there any information on how this (and libffi I guess) compare to GHC's FFI in terms of performance? Is it equivalent? Once you've loaded a function with loadSymbol and are cinvoking it with various arguments, versus a plain "foreign import" of the same. (Also, I assume cinvoke corresponds to the FFI's 'unsafe' calls, i.e. if the function tries to call back into the GHC runtime then Bad Things will happen, and it'll block threads on the same 'Capability' if it runs too long?)

...

_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

-- Work is punishment for failing to procrastinate effectively.

Remi Turk

9 Mar 9 Mar

11:26 a.m.

On Tue, Mar 08, 2011 at 01:01:58PM +0100, Gábor Lehel wrote:

...

On Sun, Mar 6, 2011 at 2:38 PM, Remi Turk wrote:

...
Where? Hackage: http://hackage.haskell.org/package/cinvoke

Cheers, Remi

[1] http://www.nongnu.org/cinvoke/

Is there any information on how this (and libffi I guess) compare to GHC's FFI in terms of performance? Is it equivalent? Once you've loaded a function with loadSymbol and are cinvoking it with various arguments, versus a plain "foreign import" of the same.

Count on it having at least an order of magnitude more overhead. I did some simple test of calling the following three trivial functions (with constant arguments, and ignoring the return values, 2M times) and got the following timings: int blub0() { return 42; } int blub1(int a) { return 42; } int blub5(int a, int b, int c, int d, int e) { return 42; } Unsafe FFI Safe FFI Safe dynamic FFI CInvoke blub0 0.03 0.19 0.20 1.62 blub1 0.03 0.20 0.20 2.44 blub5 0.04 0.20 0.20 4.35 It's not that bad for functions that actually (try to) do something though. For example, trying to remove a non-existent file: unlink 3.06 3.04 3.27 7.15 If I remember correctly, libffi was slightly faster, but mostly thanks to the fact that I didn't make it exception safe yet. So if you care about performance and are able to directly use the FFI, you clearly should.

...

(Also, I assume cinvoke corresponds to the FFI's 'unsafe' calls, i.e. if the function tries to call back into the GHC runtime then Bad Things will happen, and it'll block threads on the same 'Capability' if it runs too long?)

Actually, it doesn't: Considering the rather large overhead of CInvoke itself, I just import everything 'safe'. Though to be honest I didn't actually test any callbacks into Haskell. Cheers, Remi

Gábor Lehel

11:50 a.m.

On Wed, Mar 9, 2011 at 5:26 PM, Remi Turk wrote:

...

On Tue, Mar 08, 2011 at 01:01:58PM +0100, Gábor Lehel wrote:

...
On Sun, Mar 6, 2011 at 2:38 PM, Remi Turk wrote:

...
Where? Hackage: http://hackage.haskell.org/package/cinvoke

Cheers, Remi

[1] http://www.nongnu.org/cinvoke/

Is there any information on how this (and libffi I guess) compare to GHC's FFI in terms of performance? Is it equivalent? Once you've loaded a function with loadSymbol and are cinvoking it with various arguments, versus a plain "foreign import" of the same.

Count on it having at least an order of magnitude more overhead. I did some simple test of calling the following three trivial functions (with constant arguments, and ignoring the return values, 2M times) and got the following timings:

int blub0() { return 42; } int blub1(int a) { return 42; } int blub5(int a, int b, int c, int d, int e) { return 42; }

Unsafe FFI Safe FFI Safe dynamic FFI CInvoke blub0 0.03 0.19 0.20 1.62 blub1 0.03 0.20 0.20 2.44 blub5 0.04 0.20 0.20 4.35

It's not that bad for functions that actually (try to) do something though. For example, trying to remove a non-existent file:

unlink 3.06 3.04 3.27 7.15

If I remember correctly, libffi was slightly faster, but mostly thanks to the fact that I didn't make it exception safe yet.

So if you care about performance and are able to directly use the FFI, you clearly should.

That describes my situation. Thanks! For the record, what units were your measurements in? (I notice that the overhead of safe FFI calls seems to be pretty smallish, which is also quite heartening.)

...

...
(Also, I assume cinvoke corresponds to the FFI's 'unsafe' calls, i.e. if the function tries to call back into the GHC runtime then Bad Things will happen, and it'll block threads on the same 'Capability' if it runs too long?)

Actually, it doesn't: Considering the rather large overhead of CInvoke itself, I just import everything 'safe'. Though to be honest I didn't actually test any callbacks into Haskell.

Oh, yeah, it makes sense that the safety of the calls cinvoke makes would be same as the safety under which cinvoke itself is imported. Didn't think of that.

...

Cheers, Remi

-- Work is punishment for failing to procrastinate effectively.

Remi Turk

10 Mar 10 Mar

4:19 p.m.

On Wed, Mar 09, 2011 at 05:50:12PM +0100, Gábor Lehel wrote:

...

On Wed, Mar 9, 2011 at 5:26 PM, Remi Turk wrote:

...
Count on it having at least an order of magnitude more overhead. I did some simple test of calling the following three trivial functions (with constant arguments, and ignoring the return values, 2M times) and got the following timings:

int blub0() { return 42; } int blub1(int a) { return 42; } int blub5(int a, int b, int c, int d, int e) { return 42; }

Unsafe FFI Safe FFI Safe dynamic FFI CInvoke blub0 0.03 0.19 0.20 1.62 blub1 0.03 0.20 0.20 2.44 blub5 0.04 0.20 0.20 4.35

It's not that bad for functions that actually (try to) do something though. For example, trying to remove a non-existent file:

unlink 3.06 3.04 3.27 7.15

If I remember correctly, libffi was slightly faster, but mostly thanks to the fact that I didn't make it exception safe yet.

So if you care about performance and are able to directly use the FFI, you clearly should.

That describes my situation. Thanks!

For the record, what units were your measurements in?

(I notice that the overhead of safe FFI calls seems to be pretty smallish, which is also quite heartening.)

Everything is in seconds. So for example, 2 million unsafe calls to blub0 take 0.03 seconds: ~15ns or ~42 cycles per call (including replicateM_ overhead). I just noticed in my little non-scientific benchmark that the overhead for safe calls is significantly higher when compiling with -threaded: Unsafe FFI Safe FFI Safe dynamic FFI CInvoke blub0 0.04 0.36 0.35 2.27 blub1 0.05 0.36 0.36 3.52 blub5 0.05 0.37 0.37 5.72 unlink 3.26 3.21 3.56 8.41 Groeten, Remi

5186

Age (days ago)

5190

Last active (days ago)

List overview

Download

14 comments

6 participants

participants (6)

Daniel Fischer
Felipe Almeida Lessa
Gábor Lehel
Max Bolingbroke
Remi Turk
Yves Parès