The reason for non-GHC boot dependencies

older
Cannot build `master`, tries to...

Hécate

10 Jul 2024 10 Jul '24

9:01 a.m.

Hi devs, I had a chat earlier today with someone and found myself unable to explain the reason why GHC came with boot dependencies like xhtml, that are dependencies of Haddock and HPC. Obviously, the binaries are (haskell-)dynamically linked when distributed, but what is the reason why haddock, hpc, etc can't be (haskell-)statically linked when distributed? Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them. Cheers, Hécate 1: https://upx.github.io -- Hécate ✨ 🐦: @TechnoEmpress IRC: Hecate WWW: https://glitchbra.in RUN: BSD

Show replies by date

Andreas Klebinger

10 Jul 10 Jul

4:10 p.m.

I think they could be statically linked. But those boot libraries don't change much and generally don't really cause us nor users pain so it seems like there is little reason to do so to me.

...

Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them.

I believe Ben experimented with executable compression tools in the past with little success. But there were segfaults, executables being flagged by antivirus and perhaps more issues I forgot which just made using it unrealistic at the time. But perhaps the tooling has matured in the meantime. Since our distributions are already compressed purely for *distribution* purposes I would expect the gains there to be rather slim anyway. So it's not really that we don't care about size, just that these tools seemed not reliable enough for the benefits they offer in the past. Am 10/07/2024 um 11:01 schrieb Hécate via ghc-devs:

...

Hi devs,

I had a chat earlier today with someone and found myself unable to explain the reason why GHC came with boot dependencies like xhtml, that are dependencies of Haddock and HPC.

Obviously, the binaries are (haskell-)dynamically linked when distributed, but what is the reason why haddock, hpc, etc can't be (haskell-)statically linked when distributed?

Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them.

Cheers,

Hécate

1: https://upx.github.io

Artem Pelenitsyn

4:22 p.m.

...

I think they could be statically linked. But those boot libraries don't change much and generally don't really cause us nor users pain so it seems like there is little reason to do so to me.

There once was a sizeable patch to Haddock to switch from xhtml to Lucid, and it was rejected, seemingly, solely on the grounds that Lucid can't be added to boot libraries [1]. Since then Lucid dropped the heavier part of its dependency tree, so it's maybe not an issue anymore, but my point is that there's more to this story than what you mentioned: the need for keeping Haddock and HPC's dependencies in the set of boot libraries may slow down development of those tools. [1]: https://github.com/haskell/haddock/pull/1598#issuecomment-1621765685 On Wed, Jul 10, 2024 at 12:10 PM Andreas Klebinger via ghc-devs < ghc-devs@haskell.org> wrote:

...

I think they could be statically linked. But those boot libraries don't change much and generally don't really cause us nor users pain so it seems like there is little reason to do so to me.

...
Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them.

I believe Ben experimented with executable compression tools in the past with little success. But there were segfaults, executables being flagged by antivirus and perhaps more issues I forgot which just made using it unrealistic at the time.

But perhaps the tooling has matured in the meantime.

Since our distributions are already compressed purely for *distribution* purposes I would expect the gains there to be rather slim anyway.

So it's not really that we don't care about size, just that these tools seemed not reliable enough for the benefits they offer in the past.

Am 10/07/2024 um 11:01 schrieb Hécate via ghc-devs:

...
Hi devs,

I had a chat earlier today with someone and found myself unable to explain the reason why GHC came with boot dependencies like xhtml, that are dependencies of Haddock and HPC.

Obviously, the binaries are (haskell-)dynamically linked when distributed, but what is the reason why haddock, hpc, etc can't be (haskell-)statically linked when distributed?

Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them.

Cheers,

Hécate

1: https://upx.github.io

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Hécate

4:48 p.m.

For context, the move from xhtml to lucid2 is very much in progress, for both haddock and hpc. The necessity to avoid too many third-party libraries is that in its current (and very custom) setup, dependencies are git submodules in the GHC tree. Which somewhat make sense because these dependencies have sometimes to be adjusted when they use unstable internal APIs. Le 10/07/2024 à 18:22, Artem Pelenitsyn a écrit :

...

...
I think they could be statically linked. But those boot libraries don't change much and generally don't really cause us nor users pain so it seems like there is little reason to do so to me.

There once was a sizeable patch to Haddock to switch from xhtml to Lucid, and it was rejected, seemingly, solely on the grounds that Lucid can't be added to boot libraries [1]. Since then Lucid dropped the heavier part of its dependency tree, so it's maybe not an issue anymore, but my point is that there's more to this story than what you mentioned: the need for keeping Haddock and HPC's dependencies in the set of boot libraries may slow down development of those tools.

[1]: https://github.com/haskell/haddock/pull/1598#issuecomment-1621765685

On Wed, Jul 10, 2024 at 12:10 PM Andreas Klebinger via ghc-devs wrote:

I think they could be statically linked. But those boot libraries don't change much and generally don't really cause us nor users pain so it seems like there is little reason to do so to me.

> Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them.

I believe Ben experimented with executable compression tools in the past with little success. But there were segfaults, executables being flagged by antivirus and perhaps more issues I forgot which just made using it unrealistic at the time.

But perhaps the tooling has matured in the meantime.

Since our distributions are already compressed purely for *distribution* purposes I would expect the gains there to be rather slim anyway.

So it's not really that we don't care about size, just that these tools seemed not reliable enough for the benefits they offer in the past.

Am 10/07/2024 um 11:01 schrieb Hécate via ghc-devs: > Hi devs, > > I had a chat earlier today with someone and found myself unable to > explain the reason why GHC came with boot dependencies like xhtml, > that are dependencies of Haddock and HPC. > > Obviously, the binaries are (haskell-)dynamically linked when > distributed, but what is the reason why haddock, hpc, etc can't be > (haskell-)statically linked when distributed? > > Surely the size of binaries can't be the only concern, otherwise we'd > use upx¹ on them when distributing them. > > Cheers, > > Hécate > > > 1: https://upx.github.io > _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- Hécate ✨ 🐦: @TechnoEmpress IRC: Hecate WWW:https://glitchbra.in RUN: BSD

Hécate

4:55 p.m.

Thanks a lot for the clarification Andreas. So, the present situation is that we seem to have an implicit set of packages that are readily available to users of ghc and ghci, but which serve no direct goal for the end-user, and whose presence is fairly arbitrary. In light of the recent discussions about the mental model of dependencies, cabal and GHC, I was wondering if anything was keeping us expose these packages the user does not explicitly download. I will take some time to see how to switch to (haskell-)static linking for distribution releases. Have a nice day, Hécate Le 10/07/2024 à 18:10, Andreas Klebinger via ghc-devs a écrit :

...

I think they could be statically linked. But those boot libraries don't change much and generally don't really cause us nor users pain so it seems like there is little reason to do so to me.

...
Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them.

I believe Ben experimented with executable compression tools in the past with little success. But there were segfaults, executables being flagged by antivirus and perhaps more issues I forgot which just made using it unrealistic at the time.

But perhaps the tooling has matured in the meantime.

Since our distributions are already compressed purely for *distribution* purposes I would expect the gains there to be rather slim anyway.

So it's not really that we don't care about size, just that these tools seemed not reliable enough for the benefits they offer in the past.

Am 10/07/2024 um 11:01 schrieb Hécate via ghc-devs:

...
Hi devs,

I had a chat earlier today with someone and found myself unable to explain the reason why GHC came with boot dependencies like xhtml, that are dependencies of Haddock and HPC.

Obviously, the binaries are (haskell-)dynamically linked when distributed, but what is the reason why haddock, hpc, etc can't be (haskell-)statically linked when distributed?

Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them.

Cheers,

Hécate

1: https://upx.github.io

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- Hécate ✨ 🐦: @TechnoEmpress IRC: Hecate WWW: https://glitchbra.in RUN: BSD

Matthew Pickering

11 Jul 11 Jul

8:41 a.m.

The `haddock` executable must be dynamically linked as it must compile source files. It is much more robust to use dynamic linking when loading dependencies for TH evaluation than static linking. HLS is dynamically linked for the same reason. As such we have to distribute at least shared libraries for dependencies for haddock etc. It seems therefore that we might as well advertise these libraries are available in the package database. The things that we definitely have to advertise in the package database are `Cabal` and `ghc` and dependencies. The point about which libraries are boot libraries is a separate one. If you are developing a library which is a boot library then you are restricted to a small set of dependencies. Additional dependencies introduce a large amount of friction due to the coordindation needed during a release process. Anyone is free to develop their own libraries which use whichever dependencies they wish in user-space. Cheers, Matt On Wed, Jul 10, 2024 at 5:55 PM Hécate via ghc-devs wrote:

...

Thanks a lot for the clarification Andreas.

So, the present situation is that we seem to have an implicit set of packages that are readily available to users of ghc and ghci, but which serve no direct goal for the end-user, and whose presence is fairly arbitrary.

In light of the recent discussions about the mental model of dependencies, cabal and GHC, I was wondering if anything was keeping us expose these packages the user does not explicitly download.

I will take some time to see how to switch to (haskell-)static linking for distribution releases.

Have a nice day,

Hécate

Le 10/07/2024 à 18:10, Andreas Klebinger via ghc-devs a écrit :

...
I think they could be statically linked. But those boot libraries don't change much and generally don't really cause us nor users pain so it seems like there is little reason to do so to me.

...
Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them.

I believe Ben experimented with executable compression tools in the past with little success. But there were segfaults, executables being flagged by antivirus and perhaps more issues I forgot which just made using it unrealistic at the time.

But perhaps the tooling has matured in the meantime.

Since our distributions are already compressed purely for *distribution* purposes I would expect the gains there to be rather slim anyway.

So it's not really that we don't care about size, just that these tools seemed not reliable enough for the benefits they offer in the past.

Am 10/07/2024 um 11:01 schrieb Hécate via ghc-devs:

...
Hi devs,

I had a chat earlier today with someone and found myself unable to explain the reason why GHC came with boot dependencies like xhtml, that are dependencies of Haddock and HPC.

Obviously, the binaries are (haskell-)dynamically linked when distributed, but what is the reason why haddock, hpc, etc can't be (haskell-)statically linked when distributed?

Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them.

Cheers,

Hécate

1: https://upx.github.io

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- Hécate ✨ 🐦: @TechnoEmpress IRC: Hecate WWW: https://glitchbra.in RUN: BSD

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Michael Peyton Jones

9:28 a.m.

...

The `haddock` executable must be dynamically linked as it must compile source files.

Is this still true in the post-hi-haddock world? I thought we removed the code path in cabal that needed that, so recent versions of haddock shouldn't need it either? M On Thu, Jul 11, 2024 at 9:41 AM Matthew Pickering < matthewtpickering@gmail.com> wrote:

...

The `haddock` executable must be dynamically linked as it must compile source files. It is much more robust to use dynamic linking when loading dependencies for TH evaluation than static linking. HLS is dynamically linked for the same reason.

As such we have to distribute at least shared libraries for dependencies for haddock etc. It seems therefore that we might as well advertise these libraries are available in the package database. The things that we definitely have to advertise in the package database are `Cabal` and `ghc` and dependencies.

The point about which libraries are boot libraries is a separate one. If you are developing a library which is a boot library then you are restricted to a small set of dependencies. Additional dependencies introduce a large amount of friction due to the coordindation needed during a release process. Anyone is free to develop their own libraries which use whichever dependencies they wish in user-space.

Cheers,

Matt

On Wed, Jul 10, 2024 at 5:55 PM Hécate via ghc-devs wrote:

...
Thanks a lot for the clarification Andreas.

So, the present situation is that we seem to have an implicit set of packages that are readily available to users of ghc and ghci, but which serve no direct goal for the end-user, and whose presence is fairly arbitrary.

In light of the recent discussions about the mental model of dependencies, cabal and GHC, I was wondering if anything was keeping us expose these packages the user does not explicitly download.

I will take some time to see how to switch to (haskell-)static linking for distribution releases.

Have a nice day,

Hécate

Le 10/07/2024 à 18:10, Andreas Klebinger via ghc-devs a écrit :

...
I think they could be statically linked. But those boot libraries don't change much and generally don't really cause us nor users pain so it seems like there is little reason to do so to me.

...
Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them.

I believe Ben experimented with executable compression tools in the

past

...
...
with little success. But there were segfaults, executables being flagged by antivirus and perhaps more issues I forgot which just made using it unrealistic at the time.

But perhaps the tooling has matured in the meantime.

Since our distributions are already compressed purely for *distribution* purposes I would expect the gains there to be rather slim anyway.

So it's not really that we don't care about size, just that these tools seemed not reliable enough for the benefits they offer in the past.

Am 10/07/2024 um 11:01 schrieb Hécate via ghc-devs:

...
Hi devs,

I had a chat earlier today with someone and found myself unable to explain the reason why GHC came with boot dependencies like xhtml, that are dependencies of Haddock and HPC.

Obviously, the binaries are (haskell-)dynamically linked when distributed, but what is the reason why haddock, hpc, etc can't be (haskell-)statically linked when distributed?

Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them.

Cheers,

Hécate

1: https://upx.github.io

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- Hécate ✨ 🐦: @TechnoEmpress IRC: Hecate WWW: https://glitchbra.in RUN: BSD

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Matthew Pickering

9:33 a.m.

Haddock is not migrated yet to only require interface files. Sjoerd Visscher has been working on this recently. On Thu, Jul 11, 2024 at 10:28 AM Michael Peyton Jones wrote:

...

...
The `haddock` executable must be dynamically linked as it must compile source files.

Is this still true in the post-hi-haddock world? I thought we removed the code path in cabal that needed that, so recent versions of haddock shouldn't need it either?

M

On Thu, Jul 11, 2024 at 9:41 AM Matthew Pickering wrote:

...
The `haddock` executable must be dynamically linked as it must compile source files. It is much more robust to use dynamic linking when loading dependencies for TH evaluation than static linking. HLS is dynamically linked for the same reason.

As such we have to distribute at least shared libraries for dependencies for haddock etc. It seems therefore that we might as well advertise these libraries are available in the package database. The things that we definitely have to advertise in the package database are `Cabal` and `ghc` and dependencies.

The point about which libraries are boot libraries is a separate one. If you are developing a library which is a boot library then you are restricted to a small set of dependencies. Additional dependencies introduce a large amount of friction due to the coordindation needed during a release process. Anyone is free to develop their own libraries which use whichever dependencies they wish in user-space.

Cheers,

Matt

On Wed, Jul 10, 2024 at 5:55 PM Hécate via ghc-devs wrote:

...
Thanks a lot for the clarification Andreas.

So, the present situation is that we seem to have an implicit set of packages that are readily available to users of ghc and ghci, but which serve no direct goal for the end-user, and whose presence is fairly arbitrary.

In light of the recent discussions about the mental model of dependencies, cabal and GHC, I was wondering if anything was keeping us expose these packages the user does not explicitly download.

I will take some time to see how to switch to (haskell-)static linking for distribution releases.

Have a nice day,

Hécate

Le 10/07/2024 à 18:10, Andreas Klebinger via ghc-devs a écrit :

...
I think they could be statically linked. But those boot libraries don't change much and generally don't really cause us nor users pain so it seems like there is little reason to do so to me.

...
Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them.

I believe Ben experimented with executable compression tools in the past with little success. But there were segfaults, executables being flagged by antivirus and perhaps more issues I forgot which just made using it unrealistic at the time.

But perhaps the tooling has matured in the meantime.

Since our distributions are already compressed purely for *distribution* purposes I would expect the gains there to be rather slim anyway.

So it's not really that we don't care about size, just that these tools seemed not reliable enough for the benefits they offer in the past.

Am 10/07/2024 um 11:01 schrieb Hécate via ghc-devs:

...
Hi devs,

I had a chat earlier today with someone and found myself unable to explain the reason why GHC came with boot dependencies like xhtml, that are dependencies of Haddock and HPC.

Obviously, the binaries are (haskell-)dynamically linked when distributed, but what is the reason why haddock, hpc, etc can't be (haskell-)statically linked when distributed?

Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them.

Cheers,

Hécate

1: https://upx.github.io

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- Hécate ✨ 🐦: @TechnoEmpress IRC: Hecate WWW: https://glitchbra.in RUN: BSD

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Hécate

1:06 p.m.

Ah, that is a much better explanation for the distribution of shared libraries, thank you very much Matthew! I am not sure however that I follow your reasoning that we might as well expose them, but this is not a hill I am ready to die on right now. :) Have a nice day and thank you for the clarification! Hécate Le 11/07/2024 à 10:41, Matthew Pickering a écrit :

...

The `haddock` executable must be dynamically linked as it must compile source files. It is much more robust to use dynamic linking when loading dependencies for TH evaluation than static linking. HLS is dynamically linked for the same reason.

As such we have to distribute at least shared libraries for dependencies for haddock etc. It seems therefore that we might as well advertise these libraries are available in the package database. The things that we definitely have to advertise in the package database are `Cabal` and `ghc` and dependencies.

The point about which libraries are boot libraries is a separate one. If you are developing a library which is a boot library then you are restricted to a small set of dependencies. Additional dependencies introduce a large amount of friction due to the coordindation needed during a release process. Anyone is free to develop their own libraries which use whichever dependencies they wish in user-space.

Cheers,

Matt

On Wed, Jul 10, 2024 at 5:55 PM Hécate via ghc-devs wrote:

...
Thanks a lot for the clarification Andreas.

So, the present situation is that we seem to have an implicit set of packages that are readily available to users of ghc and ghci, but which serve no direct goal for the end-user, and whose presence is fairly arbitrary.

In light of the recent discussions about the mental model of dependencies, cabal and GHC, I was wondering if anything was keeping us expose these packages the user does not explicitly download.

I will take some time to see how to switch to (haskell-)static linking for distribution releases.

Have a nice day,

Hécate

Le 10/07/2024 à 18:10, Andreas Klebinger via ghc-devs a écrit :

...
I think they could be statically linked. But those boot libraries don't change much and generally don't really cause us nor users pain so it seems like there is little reason to do so to me.

...
Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them.

I believe Ben experimented with executable compression tools in the past with little success. But there were segfaults, executables being flagged by antivirus and perhaps more issues I forgot which just made using it unrealistic at the time.

But perhaps the tooling has matured in the meantime.

Since our distributions are already compressed purely for *distribution* purposes I would expect the gains there to be rather slim anyway.

So it's not really that we don't care about size, just that these tools seemed not reliable enough for the benefits they offer in the past.

Am 10/07/2024 um 11:01 schrieb Hécate via ghc-devs:

...
Hi devs,

I had a chat earlier today with someone and found myself unable to explain the reason why GHC came with boot dependencies like xhtml, that are dependencies of Haddock and HPC.

Obviously, the binaries are (haskell-)dynamically linked when distributed, but what is the reason why haddock, hpc, etc can't be (haskell-)statically linked when distributed?

Surely the size of binaries can't be the only concern, otherwise we'd use upx¹ on them when distributing them.

Cheers,

Hécate

1: https://upx.github.io

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -- Hécate ✨ 🐦: @TechnoEmpress IRC: Hecate WWW: https://glitchbra.in RUN: BSD

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- Hécate ✨ 🐦: @TechnoEmpress IRC: Hecate WWW: https://glitchbra.in RUN: BSD

360

Age (days ago)

361

Last active (days ago)

List overview

Download

8 comments

5 participants

participants (5)

Andreas Klebinger
Artem Pelenitsyn
Hécate
Matthew Pickering
Michael Peyton Jones

The reason for non-GHC boot dependencies

tags

participants (5)