Done now. OK, unless anyone has anything else major going in, I'll look for typos closely and send it to Mihail later today. It filled up pretty fast - thanks for the haste everyone! On Tue, Apr 29, 2014 at 7:46 PM, Austin Seipp wrote:

Good idea! Definitely one of the biggest changes.

On Tue, Apr 29, 2014 at 7:35 PM, George Colpitts wrote:

...

It doesn't have anything about the dynamic linking changes made for 7.8. I think it's worth mentioning the improvements we expect to get from that. The highlights of the release notes do mention it, so maybe that suffices.

In particular, I'm hoping that it is going to fix a lot of problems with using foreign libraries such as OpenGL from ghci. I could be wrong about that though.

On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones wrote:

...

As Austin has told us, there’s a draft of the GHC Status Report for the HCAR, here:

https://ghc.haskell.org/trac/ghc/wiki/Status/May14

Have we missed out something you have been working hard on? Do take a moment to add a bullet in an appropriate place (it’s a wiki). I’d like to be sure that we are giving credit to all the appropriate people, so please help us fix that too. GHC is a team effort.

Deadline is 1 May I think.

Thanks

Simon

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

-- Regards,

Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

-- Regards, Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

Hi, On Wed, 30 Apr 2014 17:45:35 +0100, Simon Marlow writes:

Dynamic linking has been a huge headache in GHC, and it's not clear that it's an overall improvement compared with the static linker. Now that 7.8 is out of the way, it's time to have a conversation about whether we want to do dynamic linking again for 7.10, or revert to static linking. I think Austin is going to update https://ghc.haskell.org/trac/ghc/wiki/DynamicGhcPrograms, and then we'll see where we stand.

Apart from the bugs, which are steadily being fixed (hooray), I have a little concern about -dynamic-too. The whole thing is kind of a mess, that if you forget to use it, then ghci doesn't work and TH doesn't work. And cabal used it wrong (maybe now it's getting better). It also increases the compilation times. And of course I get used to it, but I wanted to ask: would it be possible to change the default compilation method to be only dynamic? If we have only dynamic objects, can we still build a static executable (I mean haskelly half static: libc and libgmp is dynamic, but the haskell libs are not shared, so as of now)? If I remember correctly when I do gcc + ld for a simple C project, it's enough to decide ld time if I'm building a shared or a static executable. But maybe I'm oversimplifying something... Gergely

On 01/05/2014 02:26, George Colpitts wrote:

To elaborate, in the past, I had a lot of problems using libraries from the ghci prompt on the Mac but I haven't tried recently.

As an example, on the web page http://www.cs.yale.edu/homes/hudak/SOE/software1.htm for the book the Haskell School of Expression it says:

Note for OS X users: running graphics applications from GHCi is no longer supported. Instead, one has to compile a graphics program using GHC in order to run it (see example/GMIExamples.lhs for an example).

I had similar problems using the Yale Euterpea music program from ghci. When I inquired I was referred to https://ghc.haskell.org/trac/ghc/ticket/4244 and https://ghc.haskell.org/trac/ghc/ticket/781. I see that the latter is now scheduled for 7.10.1

I think we just punted on all the problems and assumed that dynamic linking would fix everything. What we've found is that dynamic linking causes a whole new set of problems and complexities. Meanwhile, many of the original problems have been fixed (except #781, which is kind of unfixable, but fortunately is also quite rare). Unfortunately dynamic linking is needed to support GHCi and TH on unregisterised platforms. In 7.8, GHCi and TH now works on some platforms where it didn't before (albeit not very widely used platforms). Cheers, Simon

On Wed, Apr 30, 2014 at 1:45 PM, Simon Marlow mailto:marlowsd@gmail.com> wrote:

On 30/04/2014 01:35, George Colpitts wrote:

It doesn't have anything about the dynamic linking changes made for 7.8. I think it's worth mentioning the improvements we expect to get from that. The highlights of the release notes do mention it, so maybe that suffices.

In particular, I'm hoping that it is going to fix a lot of problems with using foreign libraries such as OpenGL from ghci. I could be wrong about that though.

I'd like to understand more about what those problems are. As a data point, at Facebook we're using static linking (I compiled GHC with DYNAMIC_GHC_PROGRAMS=NO), we're loading upwards of 50 3rd-party C++ libraries and one gigantic shared library consisting of a ton of in-house C++ code, together with all our Haskell code into GHCi, and it works perfectly. The key to using the static linker is to not use it for C++ code - you want all your external C++ code in shared libraries and load those using the system linker.

Dynamic linking has been a huge headache in GHC, and it's not clear that it's an overall improvement compared with the static linker. Now that 7.8 is out of the way, it's time to have a conversation about whether we want to do dynamic linking again for 7.10, or revert to static linking. I think Austin is going to update https://ghc.haskell.org/trac/__ghc/wiki/DynamicGhcPrograms https://ghc.haskell.org/trac/ghc/wiki/DynamicGhcPrograms, and then we'll see where we stand.

Cheers, Simon

On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com>__> wrote:

As Austin has told us, there’s a draft of the *GHC Status Report for the HCAR*, here:____

https://ghc.haskell.org/trac/__ghc/wiki/Status/May14____ https://ghc.haskell.org/trac/ghc/wiki/Status/May14____

Have we missed out something you have been working hard on? Do take a moment to add a bullet in an appropriate place (it’s a wiki). I’d like to be sure that we are giving credit to all the appropriate people, so please help us fix that too. GHC is a team effort.____

Deadline is 1 May I think.____

Thanks____

Simon____

__ __

_________________________________________________ ghc-devs mailing list ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs

_________________________________________________ ghc-devs mailing list ghc-devs@haskell.org mailto:ghc-devs@haskell.org http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs

On 01/05/14 14:48, Dominick Samperi wrote:

The problem with some graphics libraries used via FFI (and other libraries that are not thread-safe), if I understand the situation correctly, is that ghci forks a thread when it shouldn't, causing some programs to miscalculate the available stack space (because they think there is only one thread).

The new dynamic linking support and the flag -fno-ghci-sandbox fixes this problem, and I would not vote for the removal of these features.

So I understand how -fno-ghci-sandbox avoids problems with GUI libraries that use thread-local state. But how is dynamic linking involved here? What improved in GHC 7.8 relative to 7.6 for you? And could you clarify "miscalculate the available stack space"? What needs to calculate stack space? Why? C stack space? We can certainly make a smoother experience around -fno-ghci-sandbox for using GUI libraries.

It is not clear to me from Simon's original post how linking to all of those C++ libs can continue to work if dynamic linking is removed in 7.10? Perhaps I misunderstand what you mean by "revert to static linking"?

When I say "revert to static linking" I mean make GHCi static linked, and have it load Haskell code compiled for static linking using the RTS linker (like it did in 7.6). Foreign libraries would still be loaded using the system linker, as they always have been. To be clear, I'm not officially proposing that we drop dynamic linking, but I think it's worthwhile exploring the design space again, given that we know dynamic linking has been tougher than we expected. Cheers, Simon

Thanks, Dominick

On Wed, Apr 30, 2014 at 9:26 PM, George Colpitts wrote:

...

To elaborate, in the past, I had a lot of problems using libraries from the ghci prompt on the Mac but I haven't tried recently.

As an example, on the web page for the book the Haskell School of Expression it says:

Note for OS X users: running graphics applications from GHCi is no longer supported. Instead, one has to compile a graphics program using GHC in order to run it (see example/GMIExamples.lhs for an example).

I had similar problems using the Yale Euterpea music program from ghci. When I inquired I was referred to https://ghc.haskell.org/trac/ghc/ticket/4244 and https://ghc.haskell.org/trac/ghc/ticket/781. I see that the latter is now scheduled for 7.10.1

On Wed, Apr 30, 2014 at 1:45 PM, Simon Marlow wrote:

...

On 30/04/2014 01:35, George Colpitts wrote:

...

It doesn't have anything about the dynamic linking changes made for 7.8. I think it's worth mentioning the improvements we expect to get from that. The highlights of the release notes do mention it, so maybe that suffices.

In particular, I'm hoping that it is going to fix a lot of problems with using foreign libraries such as OpenGL from ghci. I could be wrong about that though.

I'd like to understand more about what those problems are. As a data point, at Facebook we're using static linking (I compiled GHC with DYNAMIC_GHC_PROGRAMS=NO), we're loading upwards of 50 3rd-party C++ libraries and one gigantic shared library consisting of a ton of in-house C++ code, together with all our Haskell code into GHCi, and it works perfectly. The key to using the static linker is to not use it for C++ code - you want all your external C++ code in shared libraries and load those using the system linker.

Dynamic linking has been a huge headache in GHC, and it's not clear that it's an overall improvement compared with the static linker. Now that 7.8 is out of the way, it's time to have a conversation about whether we want to do dynamic linking again for 7.10, or revert to static linking. I think Austin is going to update https://ghc.haskell.org/trac/ghc/wiki/DynamicGhcPrograms, and then we'll see where we stand.

Cheers, Simon

...

On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones mailto:simonpj@microsoft.com> wrote:

As Austin has told us, there's a draft of the *GHC Status Report for the HCAR*, here:____

https://ghc.haskell.org/trac/ghc/wiki/Status/May14____

Have we missed out something you have been working hard on? Do take a moment to add a bullet in an appropriate place (it's a wiki). I'd like to be sure that we are giving credit to all the appropriate people, so please help us fix that too. GHC is a team effort.____

Deadline is 1 May I think.____

Thanks____

Simon____

__ __

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org mailto:ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

The problems I was seeing with ghci prior to 7.8 are similar to the ones reported by Edward Kmett above: addresses were mapped incorrectly. This was compounded by the fact that the library used was not thread-safe, so there were two issues, which may explain why -fno-ghci-sandbox did not help. On Thu, May 1, 2014 at 8:09 PM, Dominick Samperi wrote:

It turns out that the 7.8 update fixed problems that were NOT fixed by using -fno-ghci-sandbox with earlier versions of ghci, like 7.6.3, so this flag by itself did not address the issue. I assumed (perhaps incorrectly) that a full explanation must include the new dynamic linking support.

With 7.6.3 the problem arises when using FFI and embeded R, which involves linking to its shared library libR.so. Linking seems to be successful, but the program crashes when run, with a message about C stack overflow and/or a segfault. (Note that this does not happen when the program is compiled using ghc...only ghci has the problem.)

On the stack space question, embedded R tries to detect when C stack space is exhausted by inspecting the address of variables allocated in a function call, using code something like this (in R src/main/errors.c):

void (R_CheckStack)(void) { int dummy; intptr_t usage = R_CStackDir * (R_CStackStart - (uintptr_t)&dummy; ... }

The seg fault happens with ghci 7.6.3 with or without the flag -fno-ghci-sandbox. With 7.8.2 the seg fault happens if the flag -fno-ghci-sandbox is omitted; with the flag everything works fine.

If I understand your last comment correctly linking to libR should continue to work, even if you revert to static linking of Haskell compiled code via RTS linker. Since you say this is the way things have always been, perhaps the real explanation for why 7.8 fixed the issue is that some bug was fixed along the way...

Note that R is a C library, so the C++ issues that Carter mentions are not a factor here.

Thanks, Dominick

On Thu, May 1, 2014 at 5:27 PM, Simon Marlow wrote:

...

On 01/05/14 14:48, Dominick Samperi wrote:

...

The problem with some graphics libraries used via FFI (and other libraries that are not thread-safe), if I understand the situation correctly, is that ghci forks a thread when it shouldn't, causing some programs to miscalculate the available stack space (because they think there is only one thread).

...

The new dynamic linking support and the flag -fno-ghci-sandbox fixes this problem, and I would not vote for the removal of these features.

So I understand how -fno-ghci-sandbox avoids problems with GUI libraries that use thread-local state. But how is dynamic linking involved here? What improved in GHC 7.8 relative to 7.6 for you? And could you clarify "miscalculate the available stack space"? What needs to calculate stack space? Why? C stack space?

We can certainly make a smoother experience around -fno-ghci-sandbox for using GUI libraries.

...

It is not clear to me from Simon's original post how linking to all of those C++ libs can continue to work if dynamic linking is removed in 7.10? Perhaps I misunderstand what you mean by "revert to static linking"?

When I say "revert to static linking" I mean make GHCi static linked, and have it load Haskell code compiled for static linking using the RTS linker (like it did in 7.6). Foreign libraries would still be loaded using the system linker, as they always have been.

To be clear, I'm not officially proposing that we drop dynamic linking, but I think it's worthwhile exploring the design space again, given that we know dynamic linking has been tougher than we expected.

Cheers, Simon

...

Thanks, Dominick

On Wed, Apr 30, 2014 at 9:26 PM, George Colpitts wrote:

...

To elaborate, in the past, I had a lot of problems using libraries from the ghci prompt on the Mac but I haven't tried recently.

As an example, on the web page for the book the Haskell School of Expression it says:

Note for OS X users: running graphics applications from GHCi is no longer supported. Instead, one has to compile a graphics program using GHC in order to run it (see example/GMIExamples.lhs for an example).

I had similar problems using the Yale Euterpea music program from ghci. When I inquired I was referred to https://ghc.haskell.org/trac/ghc/ticket/4244 and https://ghc.haskell.org/trac/ghc/ticket/781. I see that the latter is now scheduled for 7.10.1

On Wed, Apr 30, 2014 at 1:45 PM, Simon Marlow wrote:

...

On 30/04/2014 01:35, George Colpitts wrote:

...

It doesn't have anything about the dynamic linking changes made for 7.8. I think it's worth mentioning the improvements we expect to get from that. The highlights of the release notes do mention it, so maybe that suffices.

In particular, I'm hoping that it is going to fix a lot of problems with using foreign libraries such as OpenGL from ghci. I could be wrong about that though.

I'd like to understand more about what those problems are. As a data point, at Facebook we're using static linking (I compiled GHC with DYNAMIC_GHC_PROGRAMS=NO), we're loading upwards of 50 3rd-party C++ libraries and one gigantic shared library consisting of a ton of in-house C++ code, together with all our Haskell code into GHCi, and it works perfectly. The key to using the static linker is to not use it for C++ code - you want all your external C++ code in shared libraries and load those using the system linker.

Dynamic linking has been a huge headache in GHC, and it's not clear that it's an overall improvement compared with the static linker. Now that 7.8 is out of the way, it's time to have a conversation about whether we want to do dynamic linking again for 7.10, or revert to static linking. I think Austin is going to update https://ghc.haskell.org/trac/ghc/wiki/DynamicGhcPrograms, and then we'll see where we stand.

Cheers, Simon

...

On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones mailto:simonpj@microsoft.com> wrote:

As Austin has told us, there's a draft of the *GHC Status Report for the HCAR*, here:____

https://ghc.haskell.org/trac/ghc/wiki/Status/May14____

Have we missed out something you have been working hard on? Do take a moment to add a bullet in an appropriate place (it's a wiki). I'd like to be sure that we are giving credit to all the appropriate people, so please help us fix that too. GHC is a team effort.____

Deadline is 1 May I think.____

Thanks____

Simon____

__ __

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org mailto:ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

I downloaded HEAD and placed DYNAMIC_GHC_PROGRAMS=NO in the "quick" section of mk/build.mk (with BuildFlavour = quick), and set DYNAMIC_GHC_PROGRAMS=NO in my environment before running configure (just to be sure!). Near the end of the build I saw some messages like "Warning: vectorization failure," but the build completed. The status at the end of configure doesn't say that dynamic linking via RTS has been turned off, and I don't know how to check that this is so. Nevertheless, I checked the linking issue and it is NOT fixed with this build, so DYNAMIC_GHC_PROGRAMS=YES is required to prevent the problems reported by me and others. On Fri, May 2, 2014 at 9:09 AM, Simon Marlow wrote:

On 02/05/2014 01:09, Dominick Samperi wrote:

...

If I understand your last comment correctly linking to libR should continue to work, even if you revert to static linking of Haskell compiled code via RTS linker. Since you say this is the way things have always been, perhaps the real explanation for why 7.8 fixed the issue is that some bug was fixed along the way...

Indeed. To know for sure we would have to test 7.8 with DYNAMIC_GHC_PROGRAMS=NO with your setup - is there a way to do that?

Cheers, Simon

...

Note that R is a C library, so the C++ issues that Carter mentions are not a factor here.

Thanks, Dominick

On Thu, May 1, 2014 at 5:27 PM, Simon Marlow wrote:

...

On 01/05/14 14:48, Dominick Samperi wrote:

...

The problem with some graphics libraries used via FFI (and other libraries that are not thread-safe), if I understand the situation correctly, is that ghci forks a thread when it shouldn't, causing some programs to miscalculate the available stack space (because they think there is only one thread).

...

The new dynamic linking support and the flag -fno-ghci-sandbox fixes this problem, and I would not vote for the removal of these features.

So I understand how -fno-ghci-sandbox avoids problems with GUI libraries that use thread-local state. But how is dynamic linking involved here? What improved in GHC 7.8 relative to 7.6 for you? And could you clarify "miscalculate the available stack space"? What needs to calculate stack space? Why? C stack space?

We can certainly make a smoother experience around -fno-ghci-sandbox for using GUI libraries.

...

It is not clear to me from Simon's original post how linking to all of those C++ libs can continue to work if dynamic linking is removed in 7.10? Perhaps I misunderstand what you mean by "revert to static linking"?

When I say "revert to static linking" I mean make GHCi static linked, and have it load Haskell code compiled for static linking using the RTS linker (like it did in 7.6). Foreign libraries would still be loaded using the system linker, as they always have been.

To be clear, I'm not officially proposing that we drop dynamic linking, but I think it's worthwhile exploring the design space again, given that we know dynamic linking has been tougher than we expected.

Cheers, Simon

...

Thanks, Dominick

On Wed, Apr 30, 2014 at 9:26 PM, George Colpitts wrote:

...

To elaborate, in the past, I had a lot of problems using libraries from the ghci prompt on the Mac but I haven't tried recently.

As an example, on the web page for the book the Haskell School of Expression it says:

Note for OS X users: running graphics applications from GHCi is no longer supported. Instead, one has to compile a graphics program using GHC in order to run it (see example/GMIExamples.lhs for an example).

I had similar problems using the Yale Euterpea music program from ghci. When I inquired I was referred to https://ghc.haskell.org/trac/ghc/ticket/4244 and https://ghc.haskell.org/trac/ghc/ticket/781. I see that the latter is now scheduled for 7.10.1

On Wed, Apr 30, 2014 at 1:45 PM, Simon Marlow wrote:

...

On 30/04/2014 01:35, George Colpitts wrote: > > > > It doesn't have anything about the dynamic linking changes made for > 7.8. > I think it's worth mentioning the improvements we expect to get from > that. The highlights of the release notes do mention it, so maybe > that > suffices. > > In particular, I'm hoping that it is going to fix a lot of problems > with > using foreign libraries such as OpenGL from ghci. I could be wrong > about > that though.

I'd like to understand more about what those problems are. As a data point, at Facebook we're using static linking (I compiled GHC with DYNAMIC_GHC_PROGRAMS=NO), we're loading upwards of 50 3rd-party C++ libraries and one gigantic shared library consisting of a ton of in-house C++ code, together with all our Haskell code into GHCi, and it works perfectly. The key to using the static linker is to not use it for C++ code - you want all your external C++ code in shared libraries and load those using the system linker.

Dynamic linking has been a huge headache in GHC, and it's not clear that it's an overall improvement compared with the static linker. Now that 7.8 is out of the way, it's time to have a conversation about whether we want to do dynamic linking again for 7.10, or revert to static linking. I think Austin is going to update https://ghc.haskell.org/trac/ghc/wiki/DynamicGhcPrograms, and then we'll see where we stand.

Cheers, Simon

> > On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones > mailto:simonpj@microsoft.com> wrote: > > As Austin has told us, there's a draft of the *GHC Status > Report > for > the HCAR*, here:____ > > https://ghc.haskell.org/trac/ghc/wiki/Status/May14____ > > > Have we missed out something you have been working hard on? > Do > take a moment to add a bullet in an appropriate place (it's a > wiki). I'd like to be sure that we are giving credit to all > the > appropriate people, so please help us fix that too. GHC is a > team > effort.____ > > Deadline is 1 May I think.____ > > Thanks____ > > Simon____ > > __ __ > > > _______________________________________________ > ghc-devs mailing list > ghc-devs@haskell.org mailto:ghc-devs@haskell.org > http://www.haskell.org/mailman/listinfo/ghc-devs > > > > > > _______________________________________________ > ghc-devs mailing list > ghc-devs@haskell.org > http://www.haskell.org/mailman/listinfo/ghc-devs >

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

I posted a ticket related to this (#8371), but the example provided there has no problems today for all versions of ghci that I tested (including 7.6.3), provided -fno-ghci-sandbox is specified. So this problem was clearly related to the threads issue. Today there are problems when DYNAMIC_GHC_PROGRAMS=NO, but they happen later, and I have not yet narrowed this down to a small example. Basically, I have an Haskell app that embeds R (as in the sample code attached to the above ticket), but when it tries to do some calculations it seg faults (works fine with 7.8.2). On Fri, May 2, 2014 at 3:45 PM, Simon Marlow wrote:

Can you give me a quick summary of how to reproduce the problem? (not including the GHC build steps)

Cheers, Simon

On 02/05/14 18:18, Dominick Samperi wrote:

...

I downloaded HEAD and placed DYNAMIC_GHC_PROGRAMS=NO in the "quick" section of mk/build.mk (with BuildFlavour = quick), and set DYNAMIC_GHC_PROGRAMS=NO in my environment before running configure (just to be sure!). Near the end of the build I saw some messages like "Warning: vectorization failure," but the build completed.

The status at the end of configure doesn't say that dynamic linking via RTS has been turned off, and I don't know how to check that this is so. Nevertheless, I checked the linking issue and it is NOT fixed with this build, so DYNAMIC_GHC_PROGRAMS=YES is required to prevent the problems reported by me and others.

On Fri, May 2, 2014 at 9:09 AM, Simon Marlow wrote:

...

On 02/05/2014 01:09, Dominick Samperi wrote:

...

If I understand your last comment correctly linking to libR should continue to work, even if you revert to static linking of Haskell compiled code via RTS linker. Since you say this is the way things have always been, perhaps the real explanation for why 7.8 fixed the issue is that some bug was fixed along the way...

Indeed. To know for sure we would have to test 7.8 with DYNAMIC_GHC_PROGRAMS=NO with your setup - is there a way to do that?

Cheers, Simon

...

Note that R is a C library, so the C++ issues that Carter mentions are not a factor here.

Thanks, Dominick

On Thu, May 1, 2014 at 5:27 PM, Simon Marlow wrote:

...

On 01/05/14 14:48, Dominick Samperi wrote:

...

The problem with some graphics libraries used via FFI (and other libraries that are not thread-safe), if I understand the situation correctly, is that ghci forks a thread when it shouldn't, causing some programs to miscalculate the available stack space (because they think there is only one thread).

...

The new dynamic linking support and the flag -fno-ghci-sandbox fixes this problem, and I would not vote for the removal of these features.

So I understand how -fno-ghci-sandbox avoids problems with GUI libraries that use thread-local state. But how is dynamic linking involved here? What improved in GHC 7.8 relative to 7.6 for you? And could you clarify "miscalculate the available stack space"? What needs to calculate stack space? Why? C stack space?

We can certainly make a smoother experience around -fno-ghci-sandbox for using GUI libraries.

...

It is not clear to me from Simon's original post how linking to all of those C++ libs can continue to work if dynamic linking is removed in 7.10? Perhaps I misunderstand what you mean by "revert to static linking"?

When I say "revert to static linking" I mean make GHCi static linked, and have it load Haskell code compiled for static linking using the RTS linker (like it did in 7.6). Foreign libraries would still be loaded using the system linker, as they always have been.

To be clear, I'm not officially proposing that we drop dynamic linking, but I think it's worthwhile exploring the design space again, given that we know dynamic linking has been tougher than we expected.

Cheers, Simon

...

Thanks, Dominick

On Wed, Apr 30, 2014 at 9:26 PM, George Colpitts wrote: > > > > To elaborate, in the past, I had a lot of problems using libraries > from > the > ghci prompt on the Mac but I haven't tried recently. > > As an example, on the web page for the book the Haskell School of > Expression > it says: > > Note for OS X users: running graphics applications from GHCi is no > longer > supported. Instead, one has to compile a graphics program using GHC > in > order > to run it (see example/GMIExamples.lhs for an example). > > I had similar problems using the Yale Euterpea music program from > ghci. > When > I inquired I was referred to > https://ghc.haskell.org/trac/ghc/ticket/4244 > and https://ghc.haskell.org/trac/ghc/ticket/781. I see that the > latter > is > now scheduled for 7.10.1 > > > On Wed, Apr 30, 2014 at 1:45 PM, Simon Marlow > wrote: >> >> >> >> >> On 30/04/2014 01:35, George Colpitts wrote: >>> >>> >>> >>> >>> It doesn't have anything about the dynamic linking changes made for >>> 7.8. >>> I think it's worth mentioning the improvements we expect to get >>> from >>> that. The highlights of the release notes do mention it, so maybe >>> that >>> suffices. >>> >>> In particular, I'm hoping that it is going to fix a lot of problems >>> with >>> using foreign libraries such as OpenGL from ghci. I could be wrong >>> about >>> that though. >> >> >> >> >> >> I'd like to understand more about what those problems are. As a >> data >> point, at Facebook we're using static linking (I compiled GHC with >> DYNAMIC_GHC_PROGRAMS=NO), we're loading upwards of 50 3rd-party C++ >> libraries and one gigantic shared library consisting of a ton of >> in-house >> C++ code, together with all our Haskell code into GHCi, and it works >> perfectly. The key to using the static linker is to not use it for >> C++ >> code >> - you want all your external C++ code in shared libraries and load >> those >> using the system linker. >> >> Dynamic linking has been a huge headache in GHC, and it's not clear >> that >> it's an overall improvement compared with the static linker. Now >> that >> 7.8 >> is out of the way, it's time to have a conversation about whether we >> want to >> do dynamic linking again for 7.10, or revert to static linking. I >> think >> Austin is going to update >> https://ghc.haskell.org/trac/ghc/wiki/DynamicGhcPrograms, and then >> we'll >> see >> where we stand. >> >> Cheers, >> Simon >> >> >> >>> >>> On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones >>> mailto:simonpj@microsoft.com> wrote: >>> >>> As Austin has told us, there's a draft of the *GHC Status >>> Report >>> for >>> the HCAR*, here:____ >>> >>> https://ghc.haskell.org/trac/ghc/wiki/Status/May14____ >>> >>> >>> Have we missed out something you have been working hard on? >>> Do >>> take a moment to add a bullet in an appropriate place (it's >>> a >>> wiki). I'd like to be sure that we are giving credit to all >>> the >>> appropriate people, so please help us fix that too. GHC is >>> a >>> team >>> effort.____ >>> >>> Deadline is 1 May I think.____ >>> >>> Thanks____ >>> >>> Simon____ >>> >>> __ __ >>> >>> >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs@haskell.org mailto:ghc-devs@haskell.org >>> http://www.haskell.org/mailman/listinfo/ghc-devs >>> >>> >>> >>> >>> >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs@haskell.org >>> http://www.haskell.org/mailman/listinfo/ghc-devs >>> > > > _______________________________________________ > ghc-devs mailing list > ghc-devs@haskell.org > http://www.haskell.org/mailman/listinfo/ghc-devs >

I don't see any reason why llvm-general with the shared-llvm flag shouldn't work with a statically linked GHCi (7.6.3 or 7.8 with DYNAMIC_GHC_PROGRAMS=NO). Doesn't it work? On 03/05/14 02:12, Carter Schonwald wrote:

if theres a way i can patch llvm-general so that i can use it with llvm static linked into rather than dylinked, i'm all ears!

llvm-general has to use some C++ wrappers (that in turn use extern "C" sections) to make parts of the llvm api accessible from hasskell.

I had trouble following some of the thread earlier today, but is the suggestion to try building those wrappers with -fPIC would make them play nice?

On Fri, May 2, 2014 at 8:52 PM, Dominick Samperi mailto:djsamperi@gmail.com> wrote:

I posted a ticket related to this (#8371), but the example provided there has no problems today for all versions of ghci that I tested (including 7.6.3), provided -fno-ghci-sandbox is specified. So this problem was clearly related to the threads issue.

Today there are problems when DYNAMIC_GHC_PROGRAMS=NO, but they happen later, and I have not yet narrowed this down to a small example. Basically, I have an Haskell app that embeds R (as in the sample code attached to the above ticket), but when it tries to do some calculations it seg faults (works fine with 7.8.2).

On Fri, May 2, 2014 at 3:45 PM, Simon Marlow mailto:marlowsd@gmail.com> wrote: > Can you give me a quick summary of how to reproduce the problem? (not > including the GHC build steps) > > Cheers, > Simon > > > On 02/05/14 18:18, Dominick Samperi wrote: >> >> I downloaded HEAD and placed DYNAMIC_GHC_PROGRAMS=NO in the "quick" >> section of mk/build.mk http://build.mk (with BuildFlavour = quick), and set >> DYNAMIC_GHC_PROGRAMS=NO >> in my environment before running configure (just to be sure!). Near >> the end of the build >> I saw some messages like "Warning: vectorization failure," but the >> build completed. >> >> The status at the end of configure doesn't say that dynamic linking >> via RTS has been >> turned off, and I don't know how to check that this is so. >> Nevertheless, I checked >> the linking issue and it is NOT fixed with this build, so >> DYNAMIC_GHC_PROGRAMS=YES is required to prevent the problems >> reported by me and others. >> >> >> On Fri, May 2, 2014 at 9:09 AM, Simon Marlow mailto:marlowsd@gmail.com> wrote: >>> >>> On 02/05/2014 01:09, Dominick Samperi wrote: >>> >>>> If I understand your last comment correctly linking to libR should >>>> continue to work, even if you revert to static linking of Haskell >>>> compiled >>>> code via RTS linker. Since you say this is the way things have always >>>> been, perhaps the real explanation for why 7.8 fixed the issue is that >>>> some bug was fixed along the way... >>> >>> >>> >>> Indeed. To know for sure we would have to test 7.8 with >>> DYNAMIC_GHC_PROGRAMS=NO with your setup - is there a way to do that? >>> >>> Cheers, >>> Simon >>> >>> >>> >>>> Note that R is a C library, so the C++ issues that Carter mentions are >>>> not a factor here. >>>> >>>> Thanks, >>>> Dominick >>>> >>>> On Thu, May 1, 2014 at 5:27 PM, Simon Marlow mailto:marlowsd@gmail.com> wrote: >>>>> >>>>> >>>>> On 01/05/14 14:48, Dominick Samperi wrote: >>>>>> >>>>>> >>>>>> >>>>>> The problem with some graphics libraries used via FFI (and other >>>>>> libraries >>>>>> that are not thread-safe), if I understand the situation correctly, is >>>>>> that ghci >>>>>> forks a thread when it shouldn't, causing some programs to >>>>>> miscalculate >>>>>> the available stack space (because they think there is only one >>>>>> thread). >>>>> >>>>> >>>>> >>>>>> >>>>>> >>>>>> The new dynamic linking support and the flag -fno-ghci-sandbox fixes >>>>>> this problem, and I would not vote for the removal of these features. >>>>> >>>>> >>>>> >>>>> >>>>> So I understand how -fno-ghci-sandbox avoids problems with GUI >>>>> libraries >>>>> that use thread-local state. But how is dynamic linking involved here? >>>>> What improved in GHC 7.8 relative to 7.6 for you? And could you clarify >>>>> "miscalculate the available stack space"? What needs to calculate stack >>>>> space? Why? C stack space? >>>>> >>>>> We can certainly make a smoother experience around -fno-ghci-sandbox >>>>> for >>>>> using GUI libraries. >>>>> >>>>> >>>>>> It is not clear to me from Simon's original post how linking to all of >>>>>> those C++ libs can continue to work if dynamic linking is removed >>>>>> in 7.10? Perhaps I misunderstand what you mean by "revert to >>>>>> static linking"? >>>>> >>>>> >>>>> >>>>> >>>>> When I say "revert to static linking" I mean make GHCi static linked, >>>>> and >>>>> have it load Haskell code compiled for static linking using the RTS >>>>> linker >>>>> (like it did in 7.6). Foreign libraries would still be loaded using >>>>> the >>>>> system linker, as they always have been. >>>>> >>>>> To be clear, I'm not officially proposing that we drop dynamic linking, >>>>> but >>>>> I think it's worthwhile exploring the design space again, given that we >>>>> know >>>>> dynamic linking has been tougher than we expected. >>>>> >>>>> Cheers, >>>>> Simon >>>>> >>>>> >>>>> >>>>>> Thanks, >>>>>> Dominick >>>>>> >>>>>> >>>>>> On Wed, Apr 30, 2014 at 9:26 PM, George Colpitts >>>>>> mailto:george.colpitts@gmail.com> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> To elaborate, in the past, I had a lot of problems using libraries >>>>>>> from >>>>>>> the >>>>>>> ghci prompt on the Mac but I haven't tried recently. >>>>>>> >>>>>>> As an example, on the web page for the book the Haskell School of >>>>>>> Expression >>>>>>> it says: >>>>>>> >>>>>>> Note for OS X users: running graphics applications from GHCi is no >>>>>>> longer >>>>>>> supported. Instead, one has to compile a graphics program using GHC >>>>>>> in >>>>>>> order >>>>>>> to run it (see example/GMIExamples.lhs for an example). >>>>>>> >>>>>>> I had similar problems using the Yale Euterpea music program from >>>>>>> ghci. >>>>>>> When >>>>>>> I inquired I was referred to >>>>>>> https://ghc.haskell.org/trac/ghc/ticket/4244 >>>>>>> and https://ghc.haskell.org/trac/ghc/ticket/781. I see that the >>>>>>> latter >>>>>>> is >>>>>>> now scheduled for 7.10.1 >>>>>>> >>>>>>> >>>>>>> On Wed, Apr 30, 2014 at 1:45 PM, Simon Marlow mailto:marlowsd@gmail.com> >>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 30/04/2014 01:35, George Colpitts wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> It doesn't have anything about the dynamic linking changes made for >>>>>>>>> 7.8. >>>>>>>>> I think it's worth mentioning the improvements we expect to get >>>>>>>>> from >>>>>>>>> that. The highlights of the release notes do mention it, so maybe >>>>>>>>> that >>>>>>>>> suffices. >>>>>>>>> >>>>>>>>> In particular, I'm hoping that it is going to fix a lot of problems >>>>>>>>> with >>>>>>>>> using foreign libraries such as OpenGL from ghci. I could be wrong >>>>>>>>> about >>>>>>>>> that though. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I'd like to understand more about what those problems are. As a >>>>>>>> data >>>>>>>> point, at Facebook we're using static linking (I compiled GHC with >>>>>>>> DYNAMIC_GHC_PROGRAMS=NO), we're loading upwards of 50 3rd-party C++ >>>>>>>> libraries and one gigantic shared library consisting of a ton of >>>>>>>> in-house >>>>>>>> C++ code, together with all our Haskell code into GHCi, and it works >>>>>>>> perfectly. The key to using the static linker is to not use it for >>>>>>>> C++ >>>>>>>> code >>>>>>>> - you want all your external C++ code in shared libraries and load >>>>>>>> those >>>>>>>> using the system linker. >>>>>>>> >>>>>>>> Dynamic linking has been a huge headache in GHC, and it's not clear >>>>>>>> that >>>>>>>> it's an overall improvement compared with the static linker. Now >>>>>>>> that >>>>>>>> 7.8 >>>>>>>> is out of the way, it's time to have a conversation about whether we >>>>>>>> want to >>>>>>>> do dynamic linking again for 7.10, or revert to static linking. I >>>>>>>> think >>>>>>>> Austin is going to update >>>>>>>> https://ghc.haskell.org/trac/ghc/wiki/DynamicGhcPrograms, and then >>>>>>>> we'll >>>>>>>> see >>>>>>>> where we stand. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Simon >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones >>>>>>>>> mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com>> wrote: >>>>>>>>> >>>>>>>>> As Austin has told us, there's a draft of the *GHC Status >>>>>>>>> Report >>>>>>>>> for >>>>>>>>> the HCAR*, here:____ >>>>>>>>> >>>>>>>>> https://ghc.haskell.org/trac/ghc/wiki/Status/May14____ >>>>>>>>> >>>>>>>>> >>>>>>>>> Have we missed out something you have been working hard on? >>>>>>>>> Do >>>>>>>>> take a moment to add a bullet in an appropriate place (it's >>>>>>>>> a >>>>>>>>> wiki). I'd like to be sure that we are giving credit to all >>>>>>>>> the >>>>>>>>> appropriate people, so please help us fix that too. GHC is >>>>>>>>> a >>>>>>>>> team >>>>>>>>> effort.____ >>>>>>>>> >>>>>>>>> Deadline is 1 May I think.____ >>>>>>>>> >>>>>>>>> Thanks____ >>>>>>>>> >>>>>>>>> Simon____ >>>>>>>>> >>>>>>>>> __ __ >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> ghc-devs mailing list >>>>>>>>> ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> >>>>>>>>> http://www.haskell.org/mailman/listinfo/ghc-devs >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> ghc-devs mailing list >>>>>>>>> ghc-devs@haskell.org mailto:ghc-devs@haskell.org >>>>>>>>> http://www.haskell.org/mailman/listinfo/ghc-devs >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> ghc-devs mailing list >>>>>>> ghc-devs@haskell.org mailto:ghc-devs@haskell.org >>>>>>> http://www.haskell.org/mailman/listinfo/ghc-devs >>>>>>> >>>>> >>> > _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org mailto:ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

On 05/05/14 21:58, Carter Schonwald wrote:

no, i was saying LLVM-General when static linked to llvm doesnt work. I wasnt talking about ghc being dynamic or static

I believe you that it doesn't work. But I think that if you use a dynamically-linked llvm (i.e. the shared-llvm flag), it *should* work, even with 7.6.3. What goes wrong in that case? Cheers, Simon

merely that theres no way to get llvm-general to work on ghci in 7.6 afaik

On Mon, May 5, 2014 at 4:54 PM, Simon Marlow mailto:marlowsd@gmail.com> wrote:

I don't see any reason why llvm-general with the shared-llvm flag shouldn't work with a statically linked GHCi (7.6.3 or 7.8 with DYNAMIC_GHC_PROGRAMS=NO). Doesn't it work?

On 03/05/14 02:12, Carter Schonwald wrote:

if theres a way i can patch llvm-general so that i can use it with llvm static linked into rather than dylinked, i'm all ears!

llvm-general has to use some C++ wrappers (that in turn use extern "C" sections) to make parts of the llvm api accessible from hasskell.

I had trouble following some of the thread earlier today, but is the suggestion to try building those wrappers with -fPIC would make them play nice?

On Fri, May 2, 2014 at 8:52 PM, Dominick Samperi mailto:djsamperi@gmail.com mailto:djsamperi@gmail.com>> wrote:

I posted a ticket related to this (#8371), but the example provided there has no problems today for all versions of ghci that I tested (including 7.6.3), provided -fno-ghci-sandbox is specified. So this problem was clearly related to the threads issue.

Today there are problems when DYNAMIC_GHC_PROGRAMS=NO, but they happen later, and I have not yet narrowed this down to a small example. Basically, I have an Haskell app that embeds R (as in the sample code attached to the above ticket), but when it tries to do some calculations it seg faults (works fine with 7.8.2).

On Fri, May 2, 2014 at 3:45 PM, Simon Marlow mailto:marlowsd@gmail.com mailto:marlowsd@gmail.com>> wrote: > Can you give me a quick summary of how to reproduce the problem? (not > including the GHC build steps) > > Cheers, > Simon > > > On 02/05/14 18:18, Dominick Samperi wrote: >> >> I downloaded HEAD and placed DYNAMIC_GHC_PROGRAMS=NO in the "quick" >> section of mk/build.mk http://build.mk http://build.mk (with BuildFlavour =

quick), and set >> DYNAMIC_GHC_PROGRAMS=NO >> in my environment before running configure (just to be sure!). Near >> the end of the build >> I saw some messages like "Warning: vectorization failure," but the >> build completed. >> >> The status at the end of configure doesn't say that dynamic linking >> via RTS has been >> turned off, and I don't know how to check that this is so. >> Nevertheless, I checked >> the linking issue and it is NOT fixed with this build, so >> DYNAMIC_GHC_PROGRAMS=YES is required to prevent the problems >> reported by me and others. >> >> >> On Fri, May 2, 2014 at 9:09 AM, Simon Marlow mailto:marlowsd@gmail.com mailto:marlowsd@gmail.com>> wrote: >>> >>> On 02/05/2014 01:09, Dominick Samperi wrote: >>> >>>> If I understand your last comment correctly linking to libR should >>>> continue to work, even if you revert to static linking of Haskell >>>> compiled >>>> code via RTS linker. Since you say this is the way things have always >>>> been, perhaps the real explanation for why 7.8 fixed the issue is that >>>> some bug was fixed along the way... >>> >>> >>> >>> Indeed. To know for sure we would have to test 7.8 with >>> DYNAMIC_GHC_PROGRAMS=NO with your setup - is there a way to do that? >>> >>> Cheers, >>> Simon >>> >>> >>> >>>> Note that R is a C library, so the C++ issues that Carter mentions are >>>> not a factor here. >>>> >>>> Thanks, >>>> Dominick >>>> >>>> On Thu, May 1, 2014 at 5:27 PM, Simon Marlow mailto:marlowsd@gmail.com mailto:marlowsd@gmail.com>> wrote: >>>>> >>>>> >>>>> On 01/05/14 14:48, Dominick Samperi wrote: >>>>>> >>>>>> >>>>>> >>>>>> The problem with some graphics libraries used via FFI (and other >>>>>> libraries >>>>>> that are not thread-safe), if I understand the situation correctly, is >>>>>> that ghci >>>>>> forks a thread when it shouldn't, causing some programs to >>>>>> miscalculate >>>>>> the available stack space (because they think there is only one >>>>>> thread). >>>>> >>>>> >>>>> >>>>>> >>>>>> >>>>>> The new dynamic linking support and the flag -fno-ghci-sandbox fixes >>>>>> this problem, and I would not vote for the removal of these features. >>>>> >>>>> >>>>> >>>>> >>>>> So I understand how -fno-ghci-sandbox avoids problems with GUI >>>>> libraries >>>>> that use thread-local state. But how is dynamic linking involved here? >>>>> What improved in GHC 7.8 relative to 7.6 for you? And could you clarify >>>>> "miscalculate the available stack space"? What needs to calculate stack >>>>> space? Why? C stack space? >>>>> >>>>> We can certainly make a smoother experience around -fno-ghci-sandbox >>>>> for >>>>> using GUI libraries. >>>>> >>>>> >>>>>> It is not clear to me from Simon's original post how linking to all of >>>>>> those C++ libs can continue to work if dynamic linking is removed >>>>>> in 7.10? Perhaps I misunderstand what you mean by "revert to >>>>>> static linking"? >>>>> >>>>> >>>>> >>>>> >>>>> When I say "revert to static linking" I mean make GHCi static linked, >>>>> and >>>>> have it load Haskell code compiled for static linking using the RTS >>>>> linker >>>>> (like it did in 7.6). Foreign libraries would still be loaded using >>>>> the >>>>> system linker, as they always have been. >>>>> >>>>> To be clear, I'm not officially proposing that we drop dynamic linking, >>>>> but >>>>> I think it's worthwhile exploring the design space again, given that we >>>>> know >>>>> dynamic linking has been tougher than we expected. >>>>> >>>>> Cheers, >>>>> Simon >>>>> >>>>> >>>>> >>>>>> Thanks, >>>>>> Dominick >>>>>> >>>>>> >>>>>> On Wed, Apr 30, 2014 at 9:26 PM, George Colpitts >>>>>> mailto:george.colpitts@gmail.com mailto:george.colpitts@gmail.com>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> To elaborate, in the past, I had a lot of problems using libraries >>>>>>> from >>>>>>> the >>>>>>> ghci prompt on the Mac but I haven't tried recently. >>>>>>> >>>>>>> As an example, on the web page for the book the Haskell School of >>>>>>> Expression >>>>>>> it says: >>>>>>> >>>>>>> Note for OS X users: running graphics applications from GHCi is no >>>>>>> longer >>>>>>> supported. Instead, one has to compile a graphics program using GHC >>>>>>> in >>>>>>> order >>>>>>> to run it (see example/GMIExamples.lhs for an example). >>>>>>> >>>>>>> I had similar problems using the Yale Euterpea music program from >>>>>>> ghci. >>>>>>> When >>>>>>> I inquired I was referred to >>>>>>> https://ghc.haskell.org/trac/__ghc/ticket/4244 https://ghc.haskell.org/trac/ghc/ticket/4244 >>>>>>> and https://ghc.haskell.org/trac/__ghc/ticket/781 https://ghc.haskell.org/trac/ghc/ticket/781. I see that the >>>>>>> latter >>>>>>> is >>>>>>> now scheduled for 7.10.1 >>>>>>> >>>>>>> >>>>>>> On Wed, Apr 30, 2014 at 1:45 PM, Simon Marlow mailto:marlowsd@gmail.com mailto:marlowsd@gmail.com>>

>>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 30/04/2014 01:35, George Colpitts wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> It doesn't have anything about the dynamic linking changes made for >>>>>>>>> 7.8. >>>>>>>>> I think it's worth mentioning the improvements we expect to get >>>>>>>>> from >>>>>>>>> that. The highlights of the release notes do mention it, so maybe >>>>>>>>> that >>>>>>>>> suffices. >>>>>>>>> >>>>>>>>> In particular, I'm hoping that it is going to fix a lot of problems >>>>>>>>> with >>>>>>>>> using foreign libraries such as OpenGL from ghci. I could be wrong >>>>>>>>> about >>>>>>>>> that though. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I'd like to understand more about what those problems are. As a >>>>>>>> data >>>>>>>> point, at Facebook we're using static linking (I compiled GHC with >>>>>>>> DYNAMIC_GHC_PROGRAMS=NO), we're loading upwards of 50 3rd-party C++ >>>>>>>> libraries and one gigantic shared library consisting of a ton of >>>>>>>> in-house >>>>>>>> C++ code, together with all our Haskell code into GHCi, and it works >>>>>>>> perfectly. The key to using the static linker is to not use it for >>>>>>>> C++ >>>>>>>> code >>>>>>>> - you want all your external C++ code in shared libraries and load >>>>>>>> those >>>>>>>> using the system linker. >>>>>>>> >>>>>>>> Dynamic linking has been a huge headache in GHC, and it's not clear >>>>>>>> that >>>>>>>> it's an overall improvement compared with the static linker. Now >>>>>>>> that >>>>>>>> 7.8 >>>>>>>> is out of the way, it's time to have a conversation about whether we >>>>>>>> want to >>>>>>>> do dynamic linking again for 7.10, or revert to static linking. I >>>>>>>> think >>>>>>>> Austin is going to update >>>>>>>> https://ghc.haskell.org/trac/__ghc/wiki/DynamicGhcPrograms https://ghc.haskell.org/trac/ghc/wiki/DynamicGhcPrograms, and then >>>>>>>> we'll >>>>>>>> see >>>>>>>> where we stand. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Simon >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones >>>>>>>>> mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com> mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com>__>> wrote: >>>>>>>>> >>>>>>>>> As Austin has told us, there's a draft of the *GHC Status >>>>>>>>> Report >>>>>>>>> for >>>>>>>>> the HCAR*, here:____ >>>>>>>>> >>>>>>>>> https://ghc.haskell.org/trac/__ghc/wiki/Status/May14____ https://ghc.haskell.org/trac/ghc/wiki/Status/May14____ >>>>>>>>> >>>>>>>>> >>>>>>>>> Have we missed out something you have been working hard on? >>>>>>>>> Do >>>>>>>>> take a moment to add a bullet in an appropriate place (it's >>>>>>>>> a >>>>>>>>> wiki). I'd like to be sure that we are giving credit to all >>>>>>>>> the >>>>>>>>> appropriate people, so please help us fix that too. GHC is >>>>>>>>> a >>>>>>>>> team >>>>>>>>> effort.____ >>>>>>>>> >>>>>>>>> Deadline is 1 May I think.____ >>>>>>>>> >>>>>>>>> Thanks____ >>>>>>>>> >>>>>>>>> Simon____ >>>>>>>>> >>>>>>>>> __ __ >>>>>>>>> >>>>>>>>> >>>>>>>>> _________________________________________________ >>>>>>>>> ghc-devs mailing list >>>>>>>>> ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org>>

>>>>>>>>> http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _________________________________________________ >>>>>>>>> ghc-devs mailing list >>>>>>>>> ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> >>>>>>>>> http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> _________________________________________________ >>>>>>> ghc-devs mailing list >>>>>>> ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> >>>>>>> http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs >>>>>>> >>>>> >>> > _________________________________________________ ghc-devs mailing list ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs

I think the issue is really "loading static C++ libs into GHCi doesn't work properly", and that's true of all versions of GHC, including 7.8. On 05/05/14 22:08, Carter Schonwald wrote:

i don't have 7.6 setup anymore, i'll see about reproducing the issue later. it falls under the umbrella of "linking not working in ghci", But it could be that didn't try the -fshared-llvm + ghci in 7.6

On Mon, May 5, 2014 at 5:01 PM, Simon Marlow mailto:marlowsd@gmail.com> wrote:

On 05/05/14 21:58, Carter Schonwald wrote:

no, i was saying LLVM-General when static linked to llvm doesnt work. I wasnt talking about ghc being dynamic or static

I believe you that it doesn't work. But I think that if you use a dynamically-linked llvm (i.e. the shared-llvm flag), it *should* work, even with 7.6.3. What goes wrong in that case?

Cheers, Simon

merely that theres no way to get llvm-general to work on ghci in 7.6 afaik

On Mon, May 5, 2014 at 4:54 PM, Simon Marlow mailto:marlowsd@gmail.com mailto:marlowsd@gmail.com>> wrote:

I don't see any reason why llvm-general with the shared-llvm flag shouldn't work with a statically linked GHCi (7.6.3 or 7.8 with DYNAMIC_GHC_PROGRAMS=NO). Doesn't it work?

On 03/05/14 02:12, Carter Schonwald wrote:

if theres a way i can patch llvm-general so that i can use it with llvm static linked into rather than dylinked, i'm all ears!

llvm-general has to use some C++ wrappers (that in turn use extern "C" sections) to make parts of the llvm api accessible from hasskell.

I had trouble following some of the thread earlier today, but is the suggestion to try building those wrappers with -fPIC would make them play nice?

On Fri, May 2, 2014 at 8:52 PM, Dominick Samperi mailto:djsamperi@gmail.com mailto:djsamperi@gmail.com> mailto:djsamperi@gmail.com mailto:djsamperi@gmail.com>>> wrote:

I posted a ticket related to this (#8371), but the example provided there has no problems today for all versions of ghci that I tested (including 7.6.3), provided -fno-ghci-sandbox is specified. So this problem was clearly related to the threads issue.

Today there are problems when DYNAMIC_GHC_PROGRAMS=NO, but they happen later, and I have not yet narrowed this down to a small example. Basically, I have an Haskell app that embeds R (as in the sample code attached to the above ticket), but when it tries to do some calculations it seg faults (works fine with 7.8.2).

On Fri, May 2, 2014 at 3:45 PM, Simon Marlow mailto:marlowsd@gmail.com mailto:marlowsd@gmail.com> mailto:marlowsd@gmail.com mailto:marlowsd@gmail.com>>> wrote: > Can you give me a quick summary of how to reproduce the problem? (not > including the GHC build steps) > > Cheers, > Simon > > > On 02/05/14 18:18, Dominick Samperi wrote: >> >> I downloaded HEAD and placed DYNAMIC_GHC_PROGRAMS=NO in the "quick" >> section of mk/build.mk http://build.mk http://build.mk http://build.mk (with BuildFlavour =

quick), and set >> DYNAMIC_GHC_PROGRAMS=NO >> in my environment before running configure (just to be sure!). Near >> the end of the build >> I saw some messages like "Warning: vectorization failure," but the >> build completed. >> >> The status at the end of configure doesn't say that dynamic linking >> via RTS has been >> turned off, and I don't know how to check that this is so. >> Nevertheless, I checked >> the linking issue and it is NOT fixed with this build, so >> DYNAMIC_GHC_PROGRAMS=YES is required to prevent the problems >> reported by me and others. >> >> >> On Fri, May 2, 2014 at 9:09 AM, Simon Marlow mailto:marlowsd@gmail.com mailto:marlowsd@gmail.com> mailto:marlowsd@gmail.com mailto:marlowsd@gmail.com>>> wrote: >>> >>> On 02/05/2014 01:09, Dominick Samperi wrote: >>> >>>> If I understand your last comment correctly linking to libR should >>>> continue to work, even if you revert to static linking of Haskell >>>> compiled >>>> code via RTS linker. Since you say this is the way things have always >>>> been, perhaps the real explanation for why 7.8 fixed the issue is that >>>> some bug was fixed along the way... >>> >>> >>> >>> Indeed. To know for sure we would have to test 7.8 with >>> DYNAMIC_GHC_PROGRAMS=NO with your setup - is there a way to do that? >>> >>> Cheers, >>> Simon >>> >>> >>> >>>> Note that R is a C library, so the C++ issues that Carter mentions are >>>> not a factor here. >>>> >>>> Thanks, >>>> Dominick >>>> >>>> On Thu, May 1, 2014 at 5:27 PM, Simon Marlow mailto:marlowsd@gmail.com mailto:marlowsd@gmail.com> mailto:marlowsd@gmail.com mailto:marlowsd@gmail.com>>> wrote: >>>>> >>>>> >>>>> On 01/05/14 14:48, Dominick Samperi wrote: >>>>>> >>>>>> >>>>>> >>>>>> The problem with some graphics libraries used via FFI (and other >>>>>> libraries >>>>>> that are not thread-safe), if I understand the situation correctly, is >>>>>> that ghci >>>>>> forks a thread when it shouldn't, causing some programs to >>>>>> miscalculate >>>>>> the available stack space (because they think there is only one >>>>>> thread). >>>>> >>>>> >>>>> >>>>>> >>>>>> >>>>>> The new dynamic linking support and the flag -fno-ghci-sandbox fixes >>>>>> this problem, and I would not vote for the removal of these features. >>>>> >>>>> >>>>> >>>>> >>>>> So I understand how -fno-ghci-sandbox avoids problems with GUI >>>>> libraries >>>>> that use thread-local state. But how is dynamic linking involved here? >>>>> What improved in GHC 7.8 relative to 7.6 for you? And could you clarify >>>>> "miscalculate the available stack space"? What needs to calculate stack >>>>> space? Why? C stack space? >>>>> >>>>> We can certainly make a smoother experience around -fno-ghci-sandbox >>>>> for >>>>> using GUI libraries. >>>>> >>>>> >>>>>> It is not clear to me from Simon's original post how linking to all of >>>>>> those C++ libs can continue to work if dynamic linking is removed >>>>>> in 7.10? Perhaps I misunderstand what you mean by "revert to >>>>>> static linking"? >>>>> >>>>> >>>>> >>>>> >>>>> When I say "revert to static linking" I mean make GHCi static linked, >>>>> and >>>>> have it load Haskell code compiled for static linking using the RTS >>>>> linker >>>>> (like it did in 7.6). Foreign libraries would still be loaded using >>>>> the >>>>> system linker, as they always have been. >>>>> >>>>> To be clear, I'm not officially proposing that we drop dynamic linking, >>>>> but >>>>> I think it's worthwhile exploring the design space again, given that we >>>>> know >>>>> dynamic linking has been tougher than we expected. >>>>> >>>>> Cheers, >>>>> Simon >>>>> >>>>> >>>>> >>>>>> Thanks, >>>>>> Dominick >>>>>> >>>>>> >>>>>> On Wed, Apr 30, 2014 at 9:26 PM, George Colpitts >>>>>> mailto:george.colpitts@gmail.com mailto:george.colpitts@gmail.com> mailto:george.colpitts@gmail.____com

mailto:george.colpitts@gmail.com>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> To elaborate, in the past, I had a lot of problems using libraries >>>>>>> from >>>>>>> the >>>>>>> ghci prompt on the Mac but I haven't tried recently. >>>>>>> >>>>>>> As an example, on the web page for the book the Haskell School of >>>>>>> Expression >>>>>>> it says: >>>>>>> >>>>>>> Note for OS X users: running graphics applications from GHCi is no >>>>>>> longer >>>>>>> supported. Instead, one has to compile a graphics program using GHC >>>>>>> in >>>>>>> order >>>>>>> to run it (see example/GMIExamples.lhs for an example). >>>>>>> >>>>>>> I had similar problems using the Yale Euterpea music program from >>>>>>> ghci. >>>>>>> When >>>>>>> I inquired I was referred to >>>>>>> https://ghc.haskell.org/trac/____ghc/ticket/4244 https://ghc.haskell.org/trac/__ghc/ticket/4244 <https://ghc.haskell.org/trac/__ghc/ticket/4244 https://ghc.haskell.org/trac/ghc/ticket/4244> >>>>>>> and https://ghc.haskell.org/trac/____ghc/ticket/781 https://ghc.haskell.org/trac/__ghc/ticket/781 <https://ghc.haskell.org/trac/__ghc/ticket/781 https://ghc.haskell.org/trac/ghc/ticket/781>. I see that the

>>>>>>> latter >>>>>>> is >>>>>>> now scheduled for 7.10.1 >>>>>>> >>>>>>> >>>>>>> On Wed, Apr 30, 2014 at 1:45 PM, Simon Marlow mailto:marlowsd@gmail.com mailto:marlowsd@gmail.com> mailto:marlowsd@gmail.com mailto:marlowsd@gmail.com>>>

>>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 30/04/2014 01:35, George Colpitts wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> It doesn't have anything about the dynamic linking changes made for >>>>>>>>> 7.8. >>>>>>>>> I think it's worth mentioning the improvements we expect to get >>>>>>>>> from >>>>>>>>> that. The highlights of the release notes do mention it, so maybe >>>>>>>>> that >>>>>>>>> suffices. >>>>>>>>> >>>>>>>>> In particular, I'm hoping that it is going to fix a lot of problems >>>>>>>>> with >>>>>>>>> using foreign libraries such as OpenGL from ghci. I could be wrong >>>>>>>>> about >>>>>>>>> that though. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I'd like to understand more about what those problems are. As a >>>>>>>> data >>>>>>>> point, at Facebook we're using static linking (I compiled GHC with >>>>>>>> DYNAMIC_GHC_PROGRAMS=NO), we're loading upwards of 50 3rd-party C++ >>>>>>>> libraries and one gigantic shared library consisting of a ton of >>>>>>>> in-house >>>>>>>> C++ code, together with all our Haskell code into GHCi, and it works >>>>>>>> perfectly. The key to using the static linker is to not use it for >>>>>>>> C++ >>>>>>>> code >>>>>>>> - you want all your external C++ code in shared libraries and load >>>>>>>> those >>>>>>>> using the system linker. >>>>>>>> >>>>>>>> Dynamic linking has been a huge headache in GHC, and it's not clear >>>>>>>> that >>>>>>>> it's an overall improvement compared with the static linker. Now >>>>>>>> that >>>>>>>> 7.8 >>>>>>>> is out of the way, it's time to have a conversation about whether we >>>>>>>> want to >>>>>>>> do dynamic linking again for 7.10, or revert to static linking. I >>>>>>>> think >>>>>>>> Austin is going to update >>>>>>>> https://ghc.haskell.org/trac/____ghc/wiki/DynamicGhcPrograms https://ghc.haskell.org/trac/__ghc/wiki/DynamicGhcPrograms

<https://ghc.haskell.org/trac/__ghc/wiki/DynamicGhcPrograms https://ghc.haskell.org/trac/ghc/wiki/DynamicGhcPrograms>,

and then >>>>>>>> we'll >>>>>>>> see >>>>>>>> where we stand. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Simon >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones >>>>>>>>> mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com> mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com>__> mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com> mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com>__>__>> wrote: >>>>>>>>> >>>>>>>>> As Austin has told us, there's a draft of the *GHC Status >>>>>>>>> Report >>>>>>>>> for >>>>>>>>> the HCAR*, here:____ >>>>>>>>> >>>>>>>>> https://ghc.haskell.org/trac/____ghc/wiki/Status/May14____ https://ghc.haskell.org/trac/__ghc/wiki/Status/May14____

<https://ghc.haskell.org/trac/__ghc/wiki/Status/May14____ https://ghc.haskell.org/trac/ghc/wiki/Status/May14____> >>>>>>>>> >>>>>>>>> >>>>>>>>> Have we missed out something you have been working hard on? >>>>>>>>> Do >>>>>>>>> take a moment to add a bullet in an appropriate place (it's >>>>>>>>> a >>>>>>>>> wiki). I'd like to be sure that we are giving credit to all >>>>>>>>> the >>>>>>>>> appropriate people, so please help us fix that too. GHC is >>>>>>>>> a >>>>>>>>> team >>>>>>>>> effort.____ >>>>>>>>> >>>>>>>>> Deadline is 1 May I think.____ >>>>>>>>> >>>>>>>>> Thanks____ >>>>>>>>> >>>>>>>>> Simon____ >>>>>>>>> >>>>>>>>> __ __ >>>>>>>>> >>>>>>>>> >>>>>>>>> ___________________________________________________

>>>>>>>>> ghc-devs mailing list >>>>>>>>> ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org>> mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org>>__>

>>>>>>>>> http://www.haskell.org/____mailman/listinfo/ghc-devs http://www.haskell.org/__mailman/listinfo/ghc-devs <http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ___________________________________________________ >>>>>>>>> ghc-devs mailing list >>>>>>>>> ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org>> >>>>>>>>> http://www.haskell.org/____mailman/listinfo/ghc-devs http://www.haskell.org/__mailman/listinfo/ghc-devs <http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> ___________________________________________________

>>>>>>> ghc-devs mailing list >>>>>>> ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org>> >>>>>>> http://www.haskell.org/____mailman/listinfo/ghc-devs http://www.haskell.org/__mailman/listinfo/ghc-devs <http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs> >>>>>>> >>>>> >>> > ___________________________________________________

ghc-devs mailing list ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org>> http://www.haskell.org/____mailman/listinfo/ghc-devs http://www.haskell.org/__mailman/listinfo/ghc-devs <http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs>

On 03/05/14 04:15, Dominick Samperi wrote:

I'm trying to understand the dynamic linking situation with the help of https://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.8, and according to this information I need to specify DYNAMIC_GHC_PROGRAMS=YES for ghci to use the system linker. I don't understand why you suggested I test with DYNAMIC_GHC_PROGRAMS=NO?

The question I'm trying to answer is "what stops working if we use DYNAMIC_GHC_PROGRAMS=NO?". So that's why I'm interested in what happens if you set this option. Thanks for your help! Cheers, Simon

Another interesting twist is that my experience under Windows is the opposite of the negative experience described on this page. What I mean by this is more versions of ghci seem to work under Windows (correctly linking to R.dll) than work under Linux!

On Fri, May 2, 2014 at 3:45 PM, Simon Marlow wrote:

...

Can you give me a quick summary of how to reproduce the problem? (not including the GHC build steps)

Cheers, Simon

On 02/05/14 18:18, Dominick Samperi wrote:

...

I downloaded HEAD and placed DYNAMIC_GHC_PROGRAMS=NO in the "quick" section of mk/build.mk (with BuildFlavour = quick), and set DYNAMIC_GHC_PROGRAMS=NO in my environment before running configure (just to be sure!). Near the end of the build I saw some messages like "Warning: vectorization failure," but the build completed.

The status at the end of configure doesn't say that dynamic linking via RTS has been turned off, and I don't know how to check that this is so. Nevertheless, I checked the linking issue and it is NOT fixed with this build, so DYNAMIC_GHC_PROGRAMS=YES is required to prevent the problems reported by me and others.

On Fri, May 2, 2014 at 9:09 AM, Simon Marlow wrote:

...

On 02/05/2014 01:09, Dominick Samperi wrote:

...

If I understand your last comment correctly linking to libR should continue to work, even if you revert to static linking of Haskell compiled code via RTS linker. Since you say this is the way things have always been, perhaps the real explanation for why 7.8 fixed the issue is that some bug was fixed along the way...

Indeed. To know for sure we would have to test 7.8 with DYNAMIC_GHC_PROGRAMS=NO with your setup - is there a way to do that?

Cheers, Simon

...

Note that R is a C library, so the C++ issues that Carter mentions are not a factor here.

Thanks, Dominick

On Thu, May 1, 2014 at 5:27 PM, Simon Marlow wrote:

...

On 01/05/14 14:48, Dominick Samperi wrote: > > > > The problem with some graphics libraries used via FFI (and other > libraries > that are not thread-safe), if I understand the situation correctly, is > that ghci > forks a thread when it shouldn't, causing some programs to > miscalculate > the available stack space (because they think there is only one > thread).

> > > The new dynamic linking support and the flag -fno-ghci-sandbox fixes > this problem, and I would not vote for the removal of these features.

So I understand how -fno-ghci-sandbox avoids problems with GUI libraries that use thread-local state. But how is dynamic linking involved here? What improved in GHC 7.8 relative to 7.6 for you? And could you clarify "miscalculate the available stack space"? What needs to calculate stack space? Why? C stack space?

We can certainly make a smoother experience around -fno-ghci-sandbox for using GUI libraries.

> It is not clear to me from Simon's original post how linking to all of > those C++ libs can continue to work if dynamic linking is removed > in 7.10? Perhaps I misunderstand what you mean by "revert to > static linking"?

When I say "revert to static linking" I mean make GHCi static linked, and have it load Haskell code compiled for static linking using the RTS linker (like it did in 7.6). Foreign libraries would still be loaded using the system linker, as they always have been.

To be clear, I'm not officially proposing that we drop dynamic linking, but I think it's worthwhile exploring the design space again, given that we know dynamic linking has been tougher than we expected.

Cheers, Simon

> Thanks, > Dominick > > > On Wed, Apr 30, 2014 at 9:26 PM, George Colpitts > wrote: >> >> >> >> To elaborate, in the past, I had a lot of problems using libraries >> from >> the >> ghci prompt on the Mac but I haven't tried recently. >> >> As an example, on the web page for the book the Haskell School of >> Expression >> it says: >> >> Note for OS X users: running graphics applications from GHCi is no >> longer >> supported. Instead, one has to compile a graphics program using GHC >> in >> order >> to run it (see example/GMIExamples.lhs for an example). >> >> I had similar problems using the Yale Euterpea music program from >> ghci. >> When >> I inquired I was referred to >> https://ghc.haskell.org/trac/ghc/ticket/4244 >> and https://ghc.haskell.org/trac/ghc/ticket/781. I see that the >> latter >> is >> now scheduled for 7.10.1 >> >> >> On Wed, Apr 30, 2014 at 1:45 PM, Simon Marlow >> wrote: >>> >>> >>> >>> >>> On 30/04/2014 01:35, George Colpitts wrote: >>>> >>>> >>>> >>>> >>>> It doesn't have anything about the dynamic linking changes made for >>>> 7.8. >>>> I think it's worth mentioning the improvements we expect to get >>>> from >>>> that. The highlights of the release notes do mention it, so maybe >>>> that >>>> suffices. >>>> >>>> In particular, I'm hoping that it is going to fix a lot of problems >>>> with >>>> using foreign libraries such as OpenGL from ghci. I could be wrong >>>> about >>>> that though. >>> >>> >>> >>> >>> >>> I'd like to understand more about what those problems are. As a >>> data >>> point, at Facebook we're using static linking (I compiled GHC with >>> DYNAMIC_GHC_PROGRAMS=NO), we're loading upwards of 50 3rd-party C++ >>> libraries and one gigantic shared library consisting of a ton of >>> in-house >>> C++ code, together with all our Haskell code into GHCi, and it works >>> perfectly. The key to using the static linker is to not use it for >>> C++ >>> code >>> - you want all your external C++ code in shared libraries and load >>> those >>> using the system linker. >>> >>> Dynamic linking has been a huge headache in GHC, and it's not clear >>> that >>> it's an overall improvement compared with the static linker. Now >>> that >>> 7.8 >>> is out of the way, it's time to have a conversation about whether we >>> want to >>> do dynamic linking again for 7.10, or revert to static linking. I >>> think >>> Austin is going to update >>> https://ghc.haskell.org/trac/ghc/wiki/DynamicGhcPrograms, and then >>> we'll >>> see >>> where we stand. >>> >>> Cheers, >>> Simon >>> >>> >>> >>>> >>>> On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones >>>> mailto:simonpj@microsoft.com> wrote: >>>> >>>> As Austin has told us, there's a draft of the *GHC Status >>>> Report >>>> for >>>> the HCAR*, here:____ >>>> >>>> https://ghc.haskell.org/trac/ghc/wiki/Status/May14____ >>>> >>>> >>>> Have we missed out something you have been working hard on? >>>> Do >>>> take a moment to add a bullet in an appropriate place (it's >>>> a >>>> wiki). I'd like to be sure that we are giving credit to all >>>> the >>>> appropriate people, so please help us fix that too. GHC is >>>> a >>>> team >>>> effort.____ >>>> >>>> Deadline is 1 May I think.____ >>>> >>>> Thanks____ >>>> >>>> Simon____ >>>> >>>> __ __ >>>> >>>> >>>> _______________________________________________ >>>> ghc-devs mailing list >>>> ghc-devs@haskell.org mailto:ghc-devs@haskell.org >>>> http://www.haskell.org/mailman/listinfo/ghc-devs >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> ghc-devs mailing list >>>> ghc-devs@haskell.org >>>> http://www.haskell.org/mailman/listinfo/ghc-devs >>>> >> >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs@haskell.org >> http://www.haskell.org/mailman/listinfo/ghc-devs >>

On 07/05/2014 05:11, Dominick Samperi wrote:

OK, so if I understand correctly, there is no point trying to narrow down my issue further as it is not supposed to work with DYNAMIC_GHC_PROGRAMS=NO (because this disables use of the system linker).

On the contrary, if the external C++ code that you depend on is in a shared library or DLL then DYNAMIC_GHC_PROGRAMS=NO should work fine.

Furthermore, if the plan to disable some form of GHC dynamic linkage is carried out, this should not affect how external C/C++ libs are linked to (using the system linker).

Correct. (but there's no plan at the moment)

From other comments in this thread it appears that there may be problems linking to C++ libs that require static initialization.

Only when both (a) the C++ code is in the same library as the Haskell code (b) DYNAMIC_GHC_PROGRAMS=NO And even then, GHC 7.8 has support for running static initializers in the linker so it might work now. Cheers, Simon

On Mon, May 5, 2014 at 4:58 PM, Simon Marlow wrote:

...

On 03/05/14 04:15, Dominick Samperi wrote:

...

I'm trying to understand the dynamic linking situation with the help of https://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.8, and according to this information I need to specify DYNAMIC_GHC_PROGRAMS=YES for ghci to use the system linker. I don't understand why you suggested I test with DYNAMIC_GHC_PROGRAMS=NO?

The question I'm trying to answer is "what stops working if we use DYNAMIC_GHC_PROGRAMS=NO?". So that's why I'm interested in what happens if you set this option.

Thanks for your help!

Cheers, Simon

...

Another interesting twist is that my experience under Windows is the opposite of the negative experience described on this page. What I mean by this is more versions of ghci seem to work under Windows (correctly linking to R.dll) than work under Linux!

On Fri, May 2, 2014 at 3:45 PM, Simon Marlow wrote:

...

Can you give me a quick summary of how to reproduce the problem? (not including the GHC build steps)

Cheers, Simon

On 02/05/14 18:18, Dominick Samperi wrote:

...

I downloaded HEAD and placed DYNAMIC_GHC_PROGRAMS=NO in the "quick" section of mk/build.mk (with BuildFlavour = quick), and set DYNAMIC_GHC_PROGRAMS=NO in my environment before running configure (just to be sure!). Near the end of the build I saw some messages like "Warning: vectorization failure," but the build completed.

The status at the end of configure doesn't say that dynamic linking via RTS has been turned off, and I don't know how to check that this is so. Nevertheless, I checked the linking issue and it is NOT fixed with this build, so DYNAMIC_GHC_PROGRAMS=YES is required to prevent the problems reported by me and others.

On Fri, May 2, 2014 at 9:09 AM, Simon Marlow wrote:

...

On 02/05/2014 01:09, Dominick Samperi wrote:

> If I understand your last comment correctly linking to libR should > continue to work, even if you revert to static linking of Haskell > compiled > code via RTS linker. Since you say this is the way things have always > been, perhaps the real explanation for why 7.8 fixed the issue is that > some bug was fixed along the way...

Indeed. To know for sure we would have to test 7.8 with DYNAMIC_GHC_PROGRAMS=NO with your setup - is there a way to do that?

Cheers, Simon

> Note that R is a C library, so the C++ issues that Carter mentions are > not a factor here. > > Thanks, > Dominick > > On Thu, May 1, 2014 at 5:27 PM, Simon Marlow > wrote: >> >> >> >> On 01/05/14 14:48, Dominick Samperi wrote: >>> >>> >>> >>> >>> The problem with some graphics libraries used via FFI (and other >>> libraries >>> that are not thread-safe), if I understand the situation correctly, >>> is >>> that ghci >>> forks a thread when it shouldn't, causing some programs to >>> miscalculate >>> the available stack space (because they think there is only one >>> thread). >> >> >> >> >>> >>> >>> The new dynamic linking support and the flag -fno-ghci-sandbox fixes >>> this problem, and I would not vote for the removal of these >>> features. >> >> >> >> >> >> So I understand how -fno-ghci-sandbox avoids problems with GUI >> libraries >> that use thread-local state. But how is dynamic linking involved >> here? >> What improved in GHC 7.8 relative to 7.6 for you? And could you >> clarify >> "miscalculate the available stack space"? What needs to calculate >> stack >> space? Why? C stack space? >> >> We can certainly make a smoother experience around -fno-ghci-sandbox >> for >> using GUI libraries. >> >> >>> It is not clear to me from Simon's original post how linking to all >>> of >>> those C++ libs can continue to work if dynamic linking is removed >>> in 7.10? Perhaps I misunderstand what you mean by "revert to >>> static linking"? >> >> >> >> >> >> When I say "revert to static linking" I mean make GHCi static linked, >> and >> have it load Haskell code compiled for static linking using the RTS >> linker >> (like it did in 7.6). Foreign libraries would still be loaded using >> the >> system linker, as they always have been. >> >> To be clear, I'm not officially proposing that we drop dynamic >> linking, >> but >> I think it's worthwhile exploring the design space again, given that >> we >> know >> dynamic linking has been tougher than we expected. >> >> Cheers, >> Simon >> >> >> >>> Thanks, >>> Dominick >>> >>> >>> On Wed, Apr 30, 2014 at 9:26 PM, George Colpitts >>> wrote: >>>> >>>> >>>> >>>> >>>> To elaborate, in the past, I had a lot of problems using libraries >>>> from >>>> the >>>> ghci prompt on the Mac but I haven't tried recently. >>>> >>>> As an example, on the web page for the book the Haskell School of >>>> Expression >>>> it says: >>>> >>>> Note for OS X users: running graphics applications from GHCi is no >>>> longer >>>> supported. Instead, one has to compile a graphics program using GHC >>>> in >>>> order >>>> to run it (see example/GMIExamples.lhs for an example). >>>> >>>> I had similar problems using the Yale Euterpea music program from >>>> ghci. >>>> When >>>> I inquired I was referred to >>>> https://ghc.haskell.org/trac/ghc/ticket/4244 >>>> and https://ghc.haskell.org/trac/ghc/ticket/781. I see that the >>>> latter >>>> is >>>> now scheduled for 7.10.1 >>>> >>>> >>>> On Wed, Apr 30, 2014 at 1:45 PM, Simon Marlow >>>> wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 30/04/2014 01:35, George Colpitts wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> It doesn't have anything about the dynamic linking changes made >>>>>> for >>>>>> 7.8. >>>>>> I think it's worth mentioning the improvements we expect to get >>>>>> from >>>>>> that. The highlights of the release notes do mention it, so maybe >>>>>> that >>>>>> suffices. >>>>>> >>>>>> In particular, I'm hoping that it is going to fix a lot of >>>>>> problems >>>>>> with >>>>>> using foreign libraries such as OpenGL from ghci. I could be >>>>>> wrong >>>>>> about >>>>>> that though. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> I'd like to understand more about what those problems are. As a >>>>> data >>>>> point, at Facebook we're using static linking (I compiled GHC with >>>>> DYNAMIC_GHC_PROGRAMS=NO), we're loading upwards of 50 3rd-party >>>>> C++ >>>>> libraries and one gigantic shared library consisting of a ton of >>>>> in-house >>>>> C++ code, together with all our Haskell code into GHCi, and it >>>>> works >>>>> perfectly. The key to using the static linker is to not use it >>>>> for >>>>> C++ >>>>> code >>>>> - you want all your external C++ code in shared libraries and load >>>>> those >>>>> using the system linker. >>>>> >>>>> Dynamic linking has been a huge headache in GHC, and it's not >>>>> clear >>>>> that >>>>> it's an overall improvement compared with the static linker. Now >>>>> that >>>>> 7.8 >>>>> is out of the way, it's time to have a conversation about whether >>>>> we >>>>> want to >>>>> do dynamic linking again for 7.10, or revert to static linking. I >>>>> think >>>>> Austin is going to update >>>>> https://ghc.haskell.org/trac/ghc/wiki/DynamicGhcPrograms, and then >>>>> we'll >>>>> see >>>>> where we stand. >>>>> >>>>> Cheers, >>>>> Simon >>>>> >>>>> >>>>> >>>>>> >>>>>> On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones >>>>>> mailto:simonpj@microsoft.com> wrote: >>>>>> >>>>>> As Austin has told us, there's a draft of the *GHC Status >>>>>> Report >>>>>> for >>>>>> the HCAR*, here:____ >>>>>> >>>>>> https://ghc.haskell.org/trac/ghc/wiki/Status/May14____ >>>>>> >>>>>> >>>>>> Have we missed out something you have been working hard >>>>>> on? >>>>>> Do >>>>>> take a moment to add a bullet in an appropriate place >>>>>> (it's >>>>>> a >>>>>> wiki). I'd like to be sure that we are giving credit to >>>>>> all >>>>>> the >>>>>> appropriate people, so please help us fix that too. GHC >>>>>> is >>>>>> a >>>>>> team >>>>>> effort.____ >>>>>> >>>>>> Deadline is 1 May I think.____ >>>>>> >>>>>> Thanks____ >>>>>> >>>>>> Simon____ >>>>>> >>>>>> __ __ >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> ghc-devs mailing list >>>>>> ghc-devs@haskell.org mailto:ghc-devs@haskell.org >>>>>> http://www.haskell.org/mailman/listinfo/ghc-devs >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> ghc-devs mailing list >>>>>> ghc-devs@haskell.org >>>>>> http://www.haskell.org/mailman/listinfo/ghc-devs >>>>>> >>>> >>>> >>>> _______________________________________________ >>>> ghc-devs mailing list >>>> ghc-devs@haskell.org >>>> http://www.haskell.org/mailman/listinfo/ghc-devs >>>> >>

Figured I'd make one case for dynamic linking: https://github.com/ekmett/rounded Dynamic linking is finally enabling us to build a version of MPFR bindings for Haskell for scientific/high precision computing with 7.8. I would really hate to lose it after all of these years trying to get it work, as I have a rather large edifice being built atop that platform. We tried and failed due to limitations of the old linker for almost 3 years. That said, -dynamic-too seems to cause me all sorts of problems elsewhere. ^C'ing out of a build and restarting it will often make a .o but lose the .dyn_o, leading to GHC + cabal getting confused and refusing to build until I clean. This hits me several times a day. -Edward On Thu, May 1, 2014 at 3:29 AM, Simon Peyton Jones wrote:

| Dynamic linking has been a huge headache in GHC, and it's not clear that | it's an overall improvement compared with the static linker. Now that | 7.8 is out of the way, it's time to have a conversation about whether we | want to do dynamic linking again for 7.10, or revert to static linking.

I echo this. Dynamic linking has had many un-anticipated costs and it is still very far from sorted out. It originally felt like a Fantastic Idea to give up our own linker and adopt the system linker, but it now feels to me like a black hole, endlessly sucking effort and increasing complexity.

My viewpoint is highly un-informed about details; I just watch the traffic going by. And of course it does have benefits that doubtless generate less traffic.

Simon

| | | | > | > On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones | > mailto:simonpj@microsoft.com> wrote: | > | > As Austin has told us, there's a draft of the *GHC Status Report | for | > the HCAR*, here:____ | > | > https://ghc.haskell.org/trac/ghc/wiki/Status/May14____ | > | > Have we missed out something you have been working hard on? Do | > take a moment to add a bullet in an appropriate place (it's a | > wiki). I'd like to be sure that we are giving credit to all the | > appropriate people, so please help us fix that too. GHC is a team | > effort.____ | > | > Deadline is 1 May I think.____ | > | > Thanks____ | > | > Simon____ | > | > __ __ | > | > | > _______________________________________________ | > ghc-devs mailing list | > ghc-devs@haskell.org mailto:ghc-devs@haskell.org | > http://www.haskell.org/mailman/listinfo/ghc-devs | > | > | > | > | > _______________________________________________ | > ghc-devs mailing list | > ghc-devs@haskell.org | > http://www.haskell.org/mailman/listinfo/ghc-devs | > _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

another example of a library that becomes usable in ghci with 7.8 is llvm-general. Namely, with ghc 7.8, when llvm-general is built so it dynamically links to the system's LLVM library, I can run / invoke LLVM operations in GHCi. I believe that this is due to the same reason many other libraries wouldn't work in ghci historically, namely the GHC static linker can't handle linking to the various C++ things (i forget the names of the linkage notions come up when loading C++, whatever those are) hackage.haskell.org/package/llvm-general mind you this is only when the sharedllvm flag is switched on. This of course has the down side that an llvm-general install thats suitable for interactive experimentation is not an llvm-beneral install thats suitable for distributing static linked binaries. On Thu, May 1, 2014 at 5:30 PM, Simon Marlow wrote:

On 01/05/14 15:27, Edward Kmett wrote:

...

Figured I'd make one case for dynamic linking:

https://github.com/ekmett/rounded

Dynamic linking is finally enabling us to build a version of MPFR bindings for Haskell for scientific/high precision computing with 7.8. I would really hate to lose it after all of these years trying to get it work, as I have a rather large edifice being built atop that platform. We tried and failed due to limitations of the old linker for almost 3 years.

I understand the issues with MPFR. But how is dynamic linking helping?

That said, -dynamic-too seems to cause me all sorts of problems

...

elsewhere. ^C'ing out of a build and restarting it will often make a .o but lose the .dyn_o, leading to GHC + cabal getting confused and refusing to build until I clean. This hits me several times a day.

We should fix this (or at least make it a lot less likely). Is there a ticket?

Cheers, Simon

-Edward

...

On Thu, May 1, 2014 at 3:29 AM, Simon Peyton Jones mailto:simonpj@microsoft.com> wrote:

| Dynamic linking has been a huge headache in GHC, and it's not clear that | it's an overall improvement compared with the static linker. Now that | 7.8 is out of the way, it's time to have a conversation about whether we | want to do dynamic linking again for 7.10, or revert to static linking.

I echo this. Dynamic linking has had many un-anticipated costs and it is still very far from sorted out. It originally felt like a Fantastic Idea to give up our own linker and adopt the system linker, but it now feels to me like a black hole, endlessly sucking effort and increasing complexity.

My viewpoint is highly un-informed about details; I just watch the traffic going by. And of course it does have benefits that doubtless generate less traffic.

Simon

| | | | > | > On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones | > mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com>> wrote: | > | > As Austin has told us, there's a draft of the *GHC Status Report | for | > the HCAR*, here:____ | > | > https://ghc.haskell.org/trac/ghc/wiki/Status/May14____ | > | > Have we missed out something you have been working hard on? Do | > take a moment to add a bullet in an appropriate place (it's a | > wiki). I'd like to be sure that we are giving credit to all the | > appropriate people, so please help us fix that too. GHC is a team | > effort.____ | > | > Deadline is 1 May I think.____ | > | > Thanks____ | > | > Simon____ | > | > __ __ | > | > | > _______________________________________________ | > ghc-devs mailing list | > ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org>

| > http://www.haskell.org/mailman/listinfo/ghc-devs | > | > | > | > | > _______________________________________________ | > ghc-devs mailing list | > ghc-devs@haskell.org mailto:ghc-devs@haskell.org | > http://www.haskell.org/mailman/listinfo/ghc-devs | > _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org mailto:ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

On 02/05/2014 00:46, Edward Kmett wrote:

With the old custom linker we weren't able to get our custom MPFR linked in properly on all platforms for use in ghci.

On Macs we ran into some rather interesting problems. We could get it to work for actual executables, but ghci would segfault with stuff resolved to clearly wrong addresses. If I recall correctly it may have been some kind of MachO symbol type that wasn't being resolved properly by the custom GHC linker, perhaps? We chased after it off and on for a long time to no avail.

It was complicated a great deal by having to build their library and merge it directly into ours, because we couldn't use system MPFR, due to the issues with the GHC garbage collector hooks into GMP.

I think this is the whole problem. Normally you would use an external dynamically-linked MPFR, but because you need custom changes to it, you have a local version of it which needs to get linked with your Haskell code. Presumably you have to arrange to build it with -fPIC somehow? If you have C/C++ code in the same library as your Haskell code, then dynamic linking will work better (at least on OS X, where they tend to change the linking semantics more often than ELF, and we have to duplicate that in our linker). This is a good point, but rather than try to fix that I would instead look at whether you could build your custom MPFR as a shared library, which should work fine regardless of whether GHCi is dynamically linked or not. The only issue should be passing the right -rpath so that executables can find your custom MPFR, which is something we already do for dynamic linking. Cheers, Simon

Switching to the system dynamic linker fo ghci seems to have resolved all of that effortlessly.

Dan Peebles has been talking to the MPFR folks to see if we can get them to expose enough information about the 'hidden' allocations they use that we can make them visible to GHC or have them do what our local fix does and avoid using the MPFR allocator for their hidden constant cache.

If they do that then we can actually link to the library like normal rather than link it in directly, but it isn't clear to me what would happen even with those hooks if we rolled back to something like the old custom linker.

-Edward

On Thu, May 1, 2014 at 5:30 PM, Simon Marlow mailto:marlowsd@gmail.com> wrote:

On 01/05/14 15:27, Edward Kmett wrote:

Figured I'd make one case for dynamic linking:

https://github.com/ekmett/__rounded https://github.com/ekmett/rounded

Dynamic linking is finally enabling us to build a version of MPFR bindings for Haskell for scientific/high precision computing with 7.8. I would really hate to lose it after all of these years trying to get it work, as I have a rather large edifice being built atop that platform. We tried and failed due to limitations of the old linker for almost 3 years.

I understand the issues with MPFR. But how is dynamic linking helping?

That said, -dynamic-too seems to cause me all sorts of problems elsewhere. ^C'ing out of a build and restarting it will often make a .o but lose the .dyn_o, leading to GHC + cabal getting confused and refusing to build until I clean. This hits me several times a day.

We should fix this (or at least make it a lot less likely). Is there a ticket?

Cheers, Simon

-Edward

On Thu, May 1, 2014 at 3:29 AM, Simon Peyton Jones mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com>__> wrote:

| Dynamic linking has been a huge headache in GHC, and it's not clear that | it's an overall improvement compared with the static linker. Now that | 7.8 is out of the way, it's time to have a conversation about whether we | want to do dynamic linking again for 7.10, or revert to static linking.

I echo this. Dynamic linking has had many un-anticipated costs and it is still very far from sorted out. It originally felt like a Fantastic Idea to give up our own linker and adopt the system linker, but it now feels to me like a black hole, endlessly sucking effort and increasing complexity.

My viewpoint is highly un-informed about details; I just watch the traffic going by. And of course it does have benefits that doubtless generate less traffic.

Simon

| | | | > | > On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones | > mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com> mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com>__>> wrote: | > | > As Austin has told us, there's a draft of the *GHC Status Report | for | > the HCAR*, here:____ | > | > https://ghc.haskell.org/trac/__ghc/wiki/Status/May14____ https://ghc.haskell.org/trac/ghc/wiki/Status/May14____ | > | > Have we missed out something you have been working hard on? Do | > take a moment to add a bullet in an appropriate place (it's a | > wiki). I'd like to be sure that we are giving credit to all the | > appropriate people, so please help us fix that too. GHC is a team | > effort.____ | > | > Deadline is 1 May I think.____ | > | > Thanks____ | > | > Simon____ | > | > __ __ | > | > | > _________________________________________________ | > ghc-devs mailing list | > ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org>>

| > http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs | > | > | > | > | > _________________________________________________ | > ghc-devs mailing list | > ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> | > http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs | > _________________________________________________ ghc-devs mailing list ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs

On 02/05/2014 14:28, Edward Kmett wrote:

Perhaps. We actually tried that originally, but had issues about where and how to get cabal to place it. We'd need it to go somewhere installed rather than the local build dir lest it not be there when we go to use the lib, but IIRC, cabal couldn't/wouldn't tell me where it was putting the final installed version and then there is the issue of the local in place runs vs. post cabal install runs and referencing that dir from subsequent projects. If we want a transparent ’cabal install rounded` It isn't clear to me how to get there down this path, but it is entirely possible I just missed something obvious.

MPFR/rounded was just an off the cuff example. Another place where the dynamic linker really helps is external c++ libraries which should now actually get all of their initializers called in the right order when launched from ghci.

External C++ libraries work just fine with the statically linked GHCi - it uses the system linker to load them, and all the initalizers get called, in the right order, as they should. So this is where I'm confused about what the problems actually are - there seems to be this perception that GHCi didn't work with external C++ code, but as far as I know it works just fine (indeed we're doing it a lot at Facebook, so there's at least an existence proof that depending on a non-trivial amount of external C++ with a statically-linked GHCi can work). Cheers, Simon

...

Cheers, Simon

...

Switching to the system dynamic linker fo ghci seems to have resolved all of that effortlessly.

Dan Peebles has been talking to the MPFR folks to see if we can get them to expose enough information about the 'hidden' allocations they use that we can make them visible to GHC or have them do what our local fix does and avoid using the MPFR allocator for their hidden constant cache.

If they do that then we can actually link to the library like normal rather than link it in directly, but it isn't clear to me what would happen even with those hooks if we rolled back to something like the old custom linker.

-Edward

On Thu, May 1, 2014 at 5:30 PM, Simon Marlow mailto:marlowsd@gmail.com> wrote:

On 01/05/14 15:27, Edward Kmett wrote:

Figured I'd make one case for dynamic linking:

https://github.com/ekmett/__rounded https://github.com/ekmett/rounded

Dynamic linking is finally enabling us to build a version of MPFR bindings for Haskell for scientific/high precision computing with 7.8. I would really hate to lose it after all of these years trying to get it work, as I have a rather large edifice being built atop that platform. We tried and failed due to limitations of the old linker for almost 3 years.

I understand the issues with MPFR. But how is dynamic linking helping?

That said, -dynamic-too seems to cause me all sorts of problems elsewhere. ^C'ing out of a build and restarting it will often make a .o but lose the .dyn_o, leading to GHC + cabal getting confused and refusing to build until I clean. This hits me several times a day.

We should fix this (or at least make it a lot less likely). Is there a ticket?

Cheers, Simon

-Edward

On Thu, May 1, 2014 at 3:29 AM, Simon Peyton Jones mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com>__> wrote:

| Dynamic linking has been a huge headache in GHC, and it's not clear that | it's an overall improvement compared with the static linker. Now that | 7.8 is out of the way, it's time to have a conversation about whether we | want to do dynamic linking again for 7.10, or revert to static linking.

I echo this. Dynamic linking has had many un-anticipated costs and it is still very far from sorted out. It originally felt like a Fantastic Idea to give up our own linker and adopt the system linker, but it now feels to me like a black hole, endlessly sucking effort and increasing complexity.

My viewpoint is highly un-informed about details; I just watch the traffic going by. And of course it does have benefits that doubtless generate less traffic.

Simon

| | | | > | > On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones | > mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com> mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com>__>> wrote: | > | > As Austin has told us, there's a draft of the *GHC Status Report | for | > the HCAR*, here:____ | > | > https://ghc.haskell.org/trac/__ghc/wiki/Status/May14____ https://ghc.haskell.org/trac/ghc/wiki/Status/May14____ | > | > Have we missed out something you have been working hard on? Do | > take a moment to add a bullet in an appropriate place (it's a | > wiki). I'd like to be sure that we are giving credit to all the | > appropriate people, so please help us fix that too. GHC is a team | > effort.____ | > | > Deadline is 1 May I think.____ | > | > Thanks____ | > | > Simon____ | > | > __ __ | > | > | > _________________________________________________ | > ghc-devs mailing list | > ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org>>

| > http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs | > | > | > | > | > _________________________________________________ | > ghc-devs mailing list | > ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> | > http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs | > _________________________________________________ ghc-devs mailing list ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> http://www.haskell.org/__mailman/listinfo/ghc-devs http://www.haskell.org/mailman/listinfo/ghc-devs

On Fri, May 2, 2014 at 10:19 AM, Edward Kmett wrote:

I may have to dig to find an example, but when I last checked it seemed that c++ libraries would load fine, but there was a problem with static initializers not getting called, when loading from ghci, so if your c++ library needed them then you'd have problems.

This has been a longstanding problem, but was reported closed in https://ghc.haskell.org/trac/ghc/ticket/5435

Come to think of it the static initializer we were trying _was_ being glued into the haskell package, not the external lib where it might have actually worked so that may be where my confusion came in. -Edward On Fri, May 2, 2014 at 3:44 PM, Simon Marlow wrote:

On 02/05/14 18:19, Edward Kmett wrote:

...

I may have to dig to find an example, but when I last checked it seemed that c++ libraries would load fine, but there was a problem with static initializers not getting called, when loading from ghci, so if your c++ library needed them then you'd have problems.

I think we're getting confused here. External C++ libraries are, and always have been, loaded by the system linker, regardless of whether GHCi itself is dynamically linked. So there's no difference in 7.8 with respect to how external C++ code is loaded.

However, if you link the C++ code into the same library as the Haskell code as you're doing with MPFR, then of course there is a difference (let's call that *internal* C++ code, for want of a better name). I think that's a rare case (I understand why you need to do it), and most people should be linking their C++ code via separate shared libraries.

An earlier version of the MPFR hackery used a static initializer to

...

replace the GHC garbage collection hook for GMP with one that played evil games to figure out if it was being called from the MPFR constant cache.

That initialized and loaded fine from ghc, but not from ghci. Our static initializer was never being called, despite the library being loaded.

If your C++ practices there forbid static initializers -- some places do -- that may be why you aren't seeing the issue.

Absolutely not, we're drowning in static initialisers :)

Cheers, Simon

-Edward

...

On Fri, May 2, 2014 at 9:47 AM, Simon Marlow mailto:marlowsd@gmail.com> wrote:

On 02/05/2014 14:28, Edward Kmett wrote: > Perhaps. We actually tried that originally, but had issues about > where and how to get cabal to place it. We'd need it to go somewhere > installed rather than the local build dir lest it not be there when > we go to use the lib, but IIRC, cabal couldn't/wouldn't tell me where > it was putting the final installed version and then there is the > issue of the local in place runs vs. post cabal install runs and > referencing that dir from subsequent projects. If we want a > transparent ’cabal install rounded` It isn't clear to me how to get > there down this path, but it is entirely possible I just missed > something obvious. > > MPFR/rounded was just an off the cuff example. Another place where > the dynamic linker really helps is external c++ libraries which > should now actually get all of their initializers called in the right > order when launched from ghci.

External C++ libraries work just fine with the statically linked GHCi - it uses the system linker to load them, and all the initalizers get called, in the right order, as they should. So this is where I'm confused about what the problems actually are - there seems to be this perception that GHCi didn't work with external C++ code, but as far as I know it works just fine (indeed we're doing it a lot at Facebook, so there's at least an existence proof that depending on a non-trivial amount of external C++ with a statically-linked GHCi can work).

Cheers, Simon

> > >> Cheers, >> Simon >> >>> Switching to the system dynamic linker fo ghci seems to have resolved >>> all of that effortlessly. >>> >>> Dan Peebles has been talking to the MPFR folks to see if we can get them >>> to expose enough information about the 'hidden' allocations they use >>> that we can make them visible to GHC or have them do what our local fix >>> does and avoid using the MPFR allocator for their hidden constant cache. >>> >>> If they do that then we can actually link to the library like normal >>> rather than link it in directly, but it isn't clear to me what would >>> happen even with those hooks if we rolled back to something like the old >>> custom linker. >>> >>> -Edward >>> >>> >>> On Thu, May 1, 2014 at 5:30 PM, Simon Marlow mailto:marlowsd@gmail.com >>> mailto:marlowsd@gmail.com>> wrote: >>> >>> On 01/05/14 15:27, Edward Kmett wrote: >>> >>> Figured I'd make one case for dynamic linking: >>> >>> https://github.com/ekmett/__rounded >>> https://github.com/ekmett/rounded >>> >>> Dynamic linking is finally enabling us to build a version of MPFR >>> bindings for Haskell for scientific/high precision computing >>> with 7.8. I >>> would really hate to lose it after all of these years trying to >>> get it >>> work, as I have a rather large edifice being built atop that >>> platform. >>> We tried and failed due to limitations of the old linker for >>> almost 3 years. >>> >>> >>> I understand the issues with MPFR. But how is dynamic linking helping? >>> >>> >>> That said, -dynamic-too seems to cause me all sorts of problems >>> elsewhere. ^C'ing out of a build and restarting it will often >>> make a .o >>> but lose the .dyn_o, leading to GHC + cabal getting confused and >>> refusing to build until I clean. This hits me several times a day. >>> >>> >>> We should fix this (or at least make it a lot less likely). Is >>> there a ticket? >>> >>> Cheers, >>> Simon >>> >>> >>> -Edward >>> >>> >>> On Thu, May 1, 2014 at 3:29 AM, Simon Peyton Jones >>> mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com> >>> mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com>>__> >>> wrote: >>> >>> | Dynamic linking has been a huge headache in GHC, and it's not >>> clear that >>> | it's an overall improvement compared with the static >>> linker. Now that >>> | 7.8 is out of the way, it's time to have a conversation about >>> whether we >>> | want to do dynamic linking again for 7.10, or revert to >>> static >>> linking. >>> >>> I echo this. Dynamic linking has had many un-anticipated >>> costs and >>> it is still very far from sorted out. It originally felt >>> like a >>> Fantastic Idea to give up our own linker and adopt the system >>> linker, but it now feels to me like a black hole, endlessly >>> sucking >>> effort and increasing complexity. >>> >>> My viewpoint is highly un-informed about details; I just >>> watch the >>> traffic going by. And of course it does have benefits that >>> doubtless generate less traffic. >>> >>> Simon >>> >>> | >>> | >>> | >>> | > >>> | > On Tue, Apr 29, 2014 at 6:13 PM, Simon Peyton Jones >>> | > mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com> >>> mailto:simonpj@microsoft.com mailto:simonpj@microsoft.com>> >>> mailto:simonpj@microsoft.com >>> mailto:simonpj@microsoft.com> mailto:simonpj@microsoft.com >>> mailto:simonpj@microsoft.com>>__>> wrote: >>> | > >>> | > As Austin has told us, there's a draft of the *GHC >>> Status Report >>> | for >>> | > the HCAR*, here:____ >>> | > >>> | > >>> https://ghc.haskell.org/trac/__ghc/wiki/Status/May14____ >>> https://ghc.haskell.org/trac/ghc/wiki/Status/May14____ >>> | > >>> | > Have we missed out something you have been working >>> hard on? Do >>> | > take a moment to add a bullet in an appropriate >>> place (it's a >>> | > wiki). I'd like to be sure that we are giving >>> credit to all the >>> | > appropriate people, so please help us fix that too. >>> GHC is >>> a team >>> | > effort.____ >>> | > >>> | > Deadline is 1 May I think.____ >>> | > >>> | > Thanks____ >>> | > >>> | > Simon____ >>> | > >>> | > __ __ >>> | > >>> | > >>> | > _________________________________________________ >>> | > ghc-devs mailing list >>> | > ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> >>> mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org>> >>> mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> >>> mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org>>> >>> >>> | > http://www.haskell.org/__ mailman/listinfo/ghc-devs >>> http://www.haskell.org/mailman/listinfo/ghc-devs >>> | > >>> | > >>> | > >>> | > >>> | > ______________________________ ___________________ >>> | > ghc-devs mailing list >>> | > ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> >>> mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org>> >>> | > http://www.haskell.org/__ mailman/listinfo/ghc-devs >>> http://www.haskell.org/mailman/listinfo/ghc-devs >>> | > >>> _________________________________________________ >>> ghc-devs mailing list >>> ghc-devs@haskell.org mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org> >>> mailto:ghc-devs@haskell.org mailto:ghc-devs@haskell.org>> >>> http://www.haskell.org/__mailman/listinfo/ghc-devs >>> http://www.haskell.org/mailman/listinfo/ghc-devs >>> >>> >>> >>>