OSX, ghci, dylib, what is the correct way?

Jason Dagit

12 Jun 2011 12 Jun '11

2:31 p.m.

Hello, I also asked this question on SO: http://stackoverflow.com/questions/6323755/osx-ghci-dylib-what-is-the-correc... I wonder where I'll get more/better answers? Read on for the question. Thanks! Jason I need to build some C code and then reference that C code via the FFI. I would like to use my binding from inside ghci on osx. On of my constraints is that I cannot just hand the C sources to ghc in the .cabal file. This is due to a limitation with ghc/cabal that may be fixed in the next release of ghc (but I want my code to work now and on older releases). See this [bug for details][1]. The gist of that bug is that the C code needs to be compiled with some Objective-C modules and ghc misinterprets those as linker scripts. I've tried many things and building the files myself with a makefile is the only thing that has worked. Really, this shouldn't be an issue though because it should be the same as if I decided to use a external C library that I didn't build myself. For the sake of this issue, let's pretend it's a separate C library that I can easily rebuild with different options. If I build the C library as a .a, then ghci comlains that it cannot open the .dylib. My first question is: Why does ghci need a .dylib and does it really use it? When I build a dylib I get a [segfault when loading the code into ghci][2]. Keep in mind, this binding works already on other platforms, both linux and windows, and the binding works fine on osx when I'm compiling instead of using ghci. This problem specific to the osx/ghci combo. In that trace above, I'm using gdb but it crashes regardless of whether I use gdb. I tracked it down to the lines that cause the crash: void _glfwClearWindowHints( void ) { memset( &_glfwLibrary.hints, 0, sizeof( _glfwLibrary.hints ) ); } The trouble maker is that memset line, well actually the problem is that when running inside ghci writing to the hints structure of `_glfwLibrary` is a memory access violation. The hints struct is simply a bunch of ints. It's very flat and simple, and thus I think the problem is an issue either with how I'm linking things or with the way ghci is loading the code. Here are the bits of my makefile that I use to build the dylib and the .a: GCCFLAGS := $(shell ghc --info | ghc -e "fmap read getContents >>= \ putStrLn . unwords . read . Data.Maybe.fromJust . lookup \ \"Gcc Linker flags\"") FRAMEWORK := -framework Cocoa -framework OpenGL GLFW_FLAG := $(GCCFLAGS) -O2 -fno-common -Iglfw/include -Iglfw/lib \ -Iglfw/lib/cocoa $(CFLAGS) all: $(BUILD_DIR)/static/libglfw.a $(BUILD_DIR)/dynamic/libglfw.dylib $(BUILD_DIR)/dynamic/libglfw.dylib: $(OBJS) $(CC) -dynamiclib -Wl,-single_module -compatibility_version 1 \ -current_version 1 \ $(GLFW_FLAG) -o $@ $(OBJS) $(GLFW_SRC) $(FRAMEWORK) $(BUILD_DIR)/static/libglfw.a: $(OBJS) ar -rcs $@ $(OBJS) Most of the flags are taken straight from the GLFW Makefile so I think they should be correct for that library. The first line looks a bit weird but it's the solution I used for this [problem][3]. Platform details: - OSX 10.6.6 - x86_64 - 4 cores - GHC version 7.0.3 installed via Haskell Platform installer - Source repo: https://github.com/dagit/GLFW-b [1]: http://hackage.haskell.org/trac/ghc/ticket/5025 [2]: http://hpaste.org/47457/segfault_in_ghci [3]: http://stackoverflow.com/questions/6217406/how-can-i-detect-if-ghc-is-set-to...

Show replies by date

Brandon Allbery

12 Jun 12 Jun

2:45 p.m.

On Sun, Jun 12, 2011 at 14:31, Jason Dagit wrote:

...

If I build the C library as a .a, then ghci comlains that it cannot open the .dylib. My first question is: Why does ghci need a .dylib and does it really use it?

Static libraries are... static. ghci would have to rebuild itself against the static archive to use it; that's how static archives work. Dynamic libraries are dynamic because they can be loaded at runtime instead of compile time.

...

When I build a dylib I get a [segfault when loading the code into ghci][2].

I believe ghci still uses its own dynamic loader instead of the standard library one, and as such it doesn't handle everything quite correctly; Mach-O is a moving target and Apple doesn't always document its changes.[1] When you compile, the system linker is used to build the executable and things work properly. [1] http://hackage.haskell.org/trac/ghc/ticket/4244

Jason Dagit

3:17 p.m.

On Sun, Jun 12, 2011 at 11:45 AM, Brandon Allbery wrote:

...

On Sun, Jun 12, 2011 at 14:31, Jason Dagit wrote:

...
If I build the C library as a .a, then ghci comlains that it cannot open the .dylib. My first question is: Why does ghci need a .dylib and does it really use it?

Static libraries are... static. ghci would have to rebuild itself against the static archive to use it; that's how static archives work. Dynamic libraries are dynamic because they can be loaded at runtime instead of compile time.

Interesting. When I use dtruss to see what files ghci opens, it definitely opens .a files. That gave me the impression it knows how to open a .a and use it at run-time.

...

...
When I build a dylib I get a [segfault when loading the code into ghci][2].

I believe ghci still uses its own dynamic loader instead of the standard library one, and as such it doesn't handle everything quite correctly; Mach-O is a moving target and Apple doesn't always document its changes.[1] When you compile, the system linker is used to build the executable and things work properly.

[1] http://hackage.haskell.org/trac/ghc/ticket/4244

It's hard for me to make sense of that ticket with so little context. This comment seems to go against what you're saying: Changed 10 months ago by igloo milestone set to 6.16.1 The best way to achieve this is probably to switch to using dynamic libraries for GHCi. Thanks, Jason

Brandon Allbery

3:32 p.m.

On Sun, Jun 12, 2011 at 15:17, Jason Dagit wrote:

...

On Sun, Jun 12, 2011 at 11:45 AM, Brandon Allbery wrote:

...
On Sun, Jun 12, 2011 at 14:31, Jason Dagit wrote:

...
If I build the C library as a .a, then ghci comlains that it cannot open the .dylib. My first question is: Why does ghci need a .dylib and does it really use it?

Static libraries are... static. ghci would have to rebuild itself against the static archive to use it; that's how static archives work. Dynamic libraries are dynamic because they can be loaded at runtime instead of compile time.

Interesting. When I use dtruss to see what files ghci opens, it definitely opens .a files. That gave me the impression it knows how to open a .a and use it at run-time.

I would have to see a trace to be certain what's going on there, but what exactly does "static library" mean if static libraries are actually dynamic, and why then are there dynamic libraries if static libraries are actually dynamic?

...

...
[1] http://hackage.haskell.org/trac/ghc/ticket/4244

It's hard for me to make sense of that ticket with so little context. This comment seems to go against what you're saying: Changed 10 months ago by igloo

milestone set to 6.16.1 The best way to achieve this is probably to switch to using dynamic libraries for GHCi.

The context for that remark is that GHC currently produces static libraries which contain references to system dynamic libraries when compiling a module; the plan is for ghc to produce dynamic libraries instead, which will simplify things considerably. (Come to think of it, this is probably why ghci is reading static libraries; they're the libHS*.a libraries and ghci is trying to find out what dynamic libraries it's supposed to load.) The translation of Igloo's comment is "we should just use the system linker the way it's supposed to be used instead of the current static+dynamic hack".

Simon Marlow

14 Jun 14 Jun

7:26 a.m.

On 12/06/2011 20:17, Jason Dagit wrote:

...

On Sun, Jun 12, 2011 at 11:45 AM, Brandon Allbery wrote:

...
On Sun, Jun 12, 2011 at 14:31, Jason Dagit wrote:

...
If I build the C library as a .a, then ghci comlains that it cannot open the .dylib. My first question is: Why does ghci need a .dylib and does it really use it?

Static libraries are... static. ghci would have to rebuild itself against the static archive to use it; that's how static archives work. Dynamic libraries are dynamic because they can be loaded at runtime instead of compile time.

Interesting. When I use dtruss to see what files ghci opens, it definitely opens .a files. That gave me the impression it knows how to open a .a and use it at run-time.

GHC as of version 7.0 can load .a files into GHCi.

...

...
...
When I build a dylib I get a [segfault when loading the code into ghci][2].

I don't know what the cause of that is - when you load a .dylib into GHCi, the normal system dynamic linker is used.

...

...
I believe ghci still uses its own dynamic loader instead of the standard library one,

Only to load .o and .a files, which the normal system dynamic linker cannot do.

...

...
[1] http://hackage.haskell.org/trac/ghc/ticket/4244

It would be nice to be able to use the system dynamic linker as an alternative to our own linker, particularly to ease porting to new platforms. However, that requires generating dynamically-linkable code from GHC (and doing so by default, which has performance implications). Cheers, Simon

Jason Dagit

12:57 p.m.

On Tue, Jun 14, 2011 at 4:26 AM, Simon Marlow wrote:

...

On 12/06/2011 20:17, Jason Dagit wrote:

...
On Sun, Jun 12, 2011 at 11:45 AM, Brandon Allbery wrote:

...
On Sun, Jun 12, 2011 at 14:31, Jason Dagit wrote:

...
If I build the C library as a .a, then ghci comlains that it cannot open the .dylib. My first question is: Why does ghci need a .dylib and does it really use it?

Static libraries are... static. ghci would have to rebuild itself against the static archive to use it; that's how static archives work. Dynamic libraries are dynamic because they can be loaded at runtime instead of compile time.

Interesting. When I use dtruss to see what files ghci opens, it definitely opens .a files. That gave me the impression it knows how to open a .a and use it at run-time.

GHC as of version 7.0 can load .a files into GHCi.

...
...
...
When I build a dylib I get a [segfault when loading the code into ghci][2].

I don't know what the cause of that is - when you load a .dylib into GHCi, the normal system dynamic linker is used.

How would you track it down? I'd really like to fix this and I don't mind debugging it, but I've tried all the things I could think of (valgrind, dtruss, gdb and printf). Valgrind didn't help because ghci couldn't read from stdin. gdb isn't so great because I was missing debug symbols for everything. I don't really believe it is an error in the RTS or the library code so it seems like gdb won't ever help here. Ideas? Thanks, Jason

Simon Marlow

15 Jun 15 Jun

5:16 a.m.

On 14/06/2011 17:57, Jason Dagit wrote:

...

On Tue, Jun 14, 2011 at 4:26 AM, Simon Marlow wrote:

...
On 12/06/2011 20:17, Jason Dagit wrote:

...
On Sun, Jun 12, 2011 at 11:45 AM, Brandon Allbery wrote:

...
On Sun, Jun 12, 2011 at 14:31, Jason Dagit wrote:

...
If I build the C library as a .a, then ghci comlains that it cannot open the .dylib. My first question is: Why does ghci need a .dylib and does it really use it?

Static libraries are... static. ghci would have to rebuild itself against the static archive to use it; that's how static archives work. Dynamic libraries are dynamic because they can be loaded at runtime instead of compile time.

Interesting. When I use dtruss to see what files ghci opens, it definitely opens .a files. That gave me the impression it knows how to open a .a and use it at run-time.

GHC as of version 7.0 can load .a files into GHCi.

...
...
...
When I build a dylib I get a [segfault when loading the code into ghci][2].

I don't know what the cause of that is - when you load a .dylib into GHCi, the normal system dynamic linker is used.

How would you track it down? I'd really like to fix this and I don't mind debugging it, but I've tried all the things I could think of (valgrind, dtruss, gdb and printf). Valgrind didn't help because ghci couldn't read from stdin. gdb isn't so great because I was missing debug symbols for everything.

I don't really believe it is an error in the RTS or the library code so it seems like gdb won't ever help here.

Ideas?

Can you build a simple .dylib, load it into GHCi and call it from Haskell? If that works, then see if you can gradually evolve the tiny example towards the real failure case and see at which point it fails. Can you build GHC yourself? If so, you can link GHC with -debug, and that will give you access to the debugging output from the linker (+RTS -Dl) and gdb will have source information about the RTS. I don't know who is generating those messages about "Reading symbols for shared libraries ...", or the ones about "Could not find object file", maybe those are generated by the system linker? Did you compile the source files with -fPIC? (I presume that's necessary, but I don't know much about OS X). Cheers, Simon

Jason Dagit

10:41 a.m.

On Wed, Jun 15, 2011 at 2:16 AM, Simon Marlow wrote:

...

On 14/06/2011 17:57, Jason Dagit wrote:

...
On Tue, Jun 14, 2011 at 4:26 AM, Simon Marlow wrote:

...
On 12/06/2011 20:17, Jason Dagit wrote:

...
On Sun, Jun 12, 2011 at 11:45 AM, Brandon Allbery wrote:

...
On Sun, Jun 12, 2011 at 14:31, Jason Dagit wrote:

...
If I build the C library as a .a, then ghci comlains that it cannot open the .dylib. My first question is: Why does ghci need a .dylib and does it really use it?

Static libraries are... static. ghci would have to rebuild itself against the static archive to use it; that's how static archives work. Dynamic libraries are dynamic because they can be loaded at runtime instead of compile time.

Interesting. When I use dtruss to see what files ghci opens, it definitely opens .a files. That gave me the impression it knows how to open a .a and use it at run-time.

GHC as of version 7.0 can load .a files into GHCi.

...
...
...
When I build a dylib I get a [segfault when loading the code into ghci][2].

I don't know what the cause of that is - when you load a .dylib into GHCi, the normal system dynamic linker is used.

How would you track it down? I'd really like to fix this and I don't mind debugging it, but I've tried all the things I could think of (valgrind, dtruss, gdb and printf). Valgrind didn't help because ghci couldn't read from stdin. gdb isn't so great because I was missing debug symbols for everything.

I don't really believe it is an error in the RTS or the library code so it seems like gdb won't ever help here.

Ideas?

Can you build a simple .dylib, load it into GHCi and call it from Haskell? If that works, then see if you can gradually evolve the tiny example towards the real failure case and see at which point it fails.

I can try this. I was hesitant to go down that road simply because this dylib does work if I remove the memset and any code that writes to that area of memory.

...

Can you build GHC yourself? If so, you can link GHC with -debug, and that will give you access to the debugging output from the linker (+RTS -Dl) and gdb will have source information about the RTS.

Building GHC myself is no problem. I usually do that on OSX anyway so that I can link with iconv from macports. This is the first time I've used the Haskell Platform version of GHC on osx.

...

I don't know who is generating those messages about "Reading symbols for shared libraries ...", or the ones about "Could not find object file", maybe those are generated by the system linker?

I'm 99% sure those are from GDB. I don't see them unless I run ghci inside gdb.

...

Did you compile the source files with -fPIC? (I presume that's necessary, but I don't know much about OS X).

I've tried with each -fPIC and -fno-common. The makefile I use is linked in the original question, but here is the link again incase you want to look at it: https://github.com/dagit/GLFW-b/blob/master/Makefile The current flag configuration matches what is in the git repo for GLFW. When I tried with -fPIC I didn't notice any difference. Thanks, Jason

Simon Marlow

10:55 a.m.

On 15/06/2011 15:41, Jason Dagit wrote:

...

On Wed, Jun 15, 2011 at 2:16 AM, Simon Marlow wrote:

...
On 14/06/2011 17:57, Jason Dagit wrote:

...
On Tue, Jun 14, 2011 at 4:26 AM, Simon Marlow wrote:

...
On 12/06/2011 20:17, Jason Dagit wrote:

...
On Sun, Jun 12, 2011 at 11:45 AM, Brandon Allbery wrote:

...
On Sun, Jun 12, 2011 at 14:31, Jason Dagit wrote: > > If I build the C library as a .a, then ghci comlains that it cannot > open the .dylib. My first question is: Why does ghci need a .dylib > and does it really use it?

Static libraries are... static. ghci would have to rebuild itself against the static archive to use it; that's how static archives work. Dynamic libraries are dynamic because they can be loaded at runtime instead of compile time.

Interesting. When I use dtruss to see what files ghci opens, it definitely opens .a files. That gave me the impression it knows how to open a .a and use it at run-time.

GHC as of version 7.0 can load .a files into GHCi.

...
...
> When I build a dylib I get a [segfault when loading the code into > ghci][2].

I don't know what the cause of that is - when you load a .dylib into GHCi, the normal system dynamic linker is used.

How would you track it down? I'd really like to fix this and I don't mind debugging it, but I've tried all the things I could think of (valgrind, dtruss, gdb and printf). Valgrind didn't help because ghci couldn't read from stdin. gdb isn't so great because I was missing debug symbols for everything.

I don't really believe it is an error in the RTS or the library code so it seems like gdb won't ever help here.

Ideas?

Can you build a simple .dylib, load it into GHCi and call it from Haskell? If that works, then see if you can gradually evolve the tiny example towards the real failure case and see at which point it fails.

I can try this. I was hesitant to go down that road simply because this dylib does work if I remove the memset and any code that writes to that area of memory.

...
Can you build GHC yourself? If so, you can link GHC with -debug, and that will give you access to the debugging output from the linker (+RTS -Dl) and gdb will have source information about the RTS.

Building GHC myself is no problem. I usually do that on OSX anyway so that I can link with iconv from macports. This is the first time I've used the Haskell Platform version of GHC on osx.

...
I don't know who is generating those messages about "Reading symbols for shared libraries ...", or the ones about "Could not find object file", maybe those are generated by the system linker?

I'm 99% sure those are from GDB. I don't see them unless I run ghci inside gdb.

...
Did you compile the source files with -fPIC? (I presume that's necessary, but I don't know much about OS X).

I've tried with each -fPIC and -fno-common. The makefile I use is linked in the original question, but here is the link again incase you want to look at it: https://github.com/dagit/GLFW-b/blob/master/Makefile

The current flag configuration matches what is in the git repo for GLFW. When I tried with -fPIC I didn't notice any difference.

The symptoms don't look quite right, but I wonder if it is related to this at all? http://hackage.haskell.org/trac/ghc/ticket/781 Cheers, Simon

Raeez Lorgat

23 Jun 23 Jun

9:07 a.m.

If anyone is curious, I've posted a possible solution at the stack overflow post: http://stackoverflow.com/questions/6323755/osx-ghci-dylib-what-is-the-correc... I'd appreciate any feedback (and/or refutations), especially regarding the commentary re: ghci and loading static archives. Excerpts from Simon Marlow's message of Wed Jun 15 16:55:05 +0200 2011:

...

On 15/06/2011 15:41, Jason Dagit wrote:

...
On Wed, Jun 15, 2011 at 2:16 AM, Simon Marlow wrote:

...
On 14/06/2011 17:57, Jason Dagit wrote:

...
On Tue, Jun 14, 2011 at 4:26 AM, Simon Marlow wrote:

...
On 12/06/2011 20:17, Jason Dagit wrote:

...
On Sun, Jun 12, 2011 at 11:45 AM, Brandon Allbery wrote: > > On Sun, Jun 12, 2011 at 14:31, Jason Dagit wrote: >> >> If I build the C library as a .a, then ghci comlains that it cannot >> open the .dylib. My first question is: Why does ghci need a .dylib >> and does it really use it? > > Static libraries are... static. ghci would have to rebuild itself > against the static archive to use it; that's how static archives work. > Dynamic libraries are dynamic because they can be loaded at runtime > instead of compile time.

Interesting. When I use dtruss to see what files ghci opens, it definitely opens .a files. That gave me the impression it knows how to open a .a and use it at run-time.

GHC as of version 7.0 can load .a files into GHCi.

...
>> When I build a dylib I get a [segfault when loading the code into >> ghci][2].

I don't know what the cause of that is - when you load a .dylib into GHCi, the normal system dynamic linker is used.

How would you track it down? I'd really like to fix this and I don't mind debugging it, but I've tried all the things I could think of (valgrind, dtruss, gdb and printf). Valgrind didn't help because ghci couldn't read from stdin. gdb isn't so great because I was missing debug symbols for everything.

I don't really believe it is an error in the RTS or the library code so it seems like gdb won't ever help here.

Ideas?

Can you build a simple .dylib, load it into GHCi and call it from Haskell? If that works, then see if you can gradually evolve the tiny example towards the real failure case and see at which point it fails.

I can try this. I was hesitant to go down that road simply because this dylib does work if I remove the memset and any code that writes to that area of memory.

...
Can you build GHC yourself? If so, you can link GHC with -debug, and that will give you access to the debugging output from the linker (+RTS -Dl) and gdb will have source information about the RTS.

Building GHC myself is no problem. I usually do that on OSX anyway so that I can link with iconv from macports. This is the first time I've used the Haskell Platform version of GHC on osx.

...
I don't know who is generating those messages about "Reading symbols for shared libraries ...", or the ones about "Could not find object file", maybe those are generated by the system linker?

I'm 99% sure those are from GDB. I don't see them unless I run ghci inside gdb.

...
Did you compile the source files with -fPIC? (I presume that's necessary, but I don't know much about OS X).

I've tried with each -fPIC and -fno-common. The makefile I use is linked in the original question, but here is the link again incase you want to look at it: https://github.com/dagit/GLFW-b/blob/master/Makefile

The current flag configuration matches what is in the git repo for GLFW. When I tried with -fPIC I didn't notice any difference.

The symptoms don't look quite right, but I wonder if it is related to this at all?

http://hackage.haskell.org/trac/ghc/ticket/781

Cheers, Simon

Jason Dagit

1:35 p.m.

On Thu, Jun 23, 2011 at 6:07 AM, Raeez Lorgat wrote:

...

If anyone is curious, I've posted a possible solution at the stack overflow post:

http://stackoverflow.com/questions/6323755/osx-ghci-dylib-what-is-the-correc...

I'd appreciate any feedback (and/or refutations), especially regarding the commentary re: ghci and loading static archives.

What you suggest works for me. I was a bit skeptical about using SO for a question that was so nitty and gritty about GHCi, but the experiment was a success. I think I'll be moving more of my questions over there in the future. Thank you so much! Jason

5084

Age (days ago)

5095

Last active (days ago)

List overview

Download

10 comments

4 participants

participants (4)

Brandon Allbery
Jason Dagit
Raeez Lorgat
Simon Marlow

OSX, ghci, dylib, what is the correct way?

tags

participants (4)