On Tue, Oct 17, 2023 at 04:54:41PM +0100, Adam Gundry wrote:

Thanks for starting this discussion, it would be good to see progress in this direction. As it happens I was discussing this question with Ben and Matt over dinner last night, and unfortunately they explained to me that it is more difficult than I naively hoped, even once wired-in and known-key things are moved to ghc-internal.

The difficulty is that, as a normal Haskell library, ghc itself will be compiled against a particular version of base. Then when Template Haskell is used (with the internal interpreter), code will be dynamically loaded into a process that already has symbols for ghc's version of base, which means it is not safe for the code to depend on a different version of base. This is rather like the situation with TH and cross-compilers.

To avoid that problem, GHC's own dependency on "base" could be indirect via a shared object with versioned symbol names and a version-specific SONAME (possibly even a private to GHC SONAME and private symbol version names). Say "libbase.so.4.19.1". The dependency on "base" in the TemplatHaskell generated code would then also need to be dynamic, allowing the two versions of base to coexist without conflict, both in turn depdent on a common version of the GHC internal libraries. This would of course somewhat complicate binary distributions, but that should be manageable. Perhaps there are less invasive (more clever) solutions? -- Viktor

"Howard B. Golden" writes:

My _very_ naive question is whether Template Haskell should have all the features of the latest language? Perhaps it could instead be limited to a simpler subset so that it could become a bolt-on to the rest of the compiler and it could be compiled and implemented independently using an older compiler and libraries.

While I think users should be able to access all of the features of the language from TemplateHaskell, I agree that most users only need a small subset. We have explored some ideas exploiting this property in #24021. While it's not trivial per se, I think there is a real opportunity here. Cheers, - Ben

the `List` type provided by `base-X.Y.Z` and `base-X.Y.Z'` may differ. This is not actually true, `List` will obviously live in `ghc-internal` and `ghc-internal` would be the same for `base-4.9` and for `base-4.7.1`. This raises the question about do we really need to depend on `base` in `template-haskell`? Perhaps we may be satisfied by small set of things that will contain `ghc-internal`?

20.10.2023 14:23, Ben Gamari writes:

Viktor Dukhovni writes:

...

On Tue, Oct 17, 2023 at 04:54:41PM +0100, Adam Gundry wrote:

...

Thanks for starting this discussion, it would be good to see progress in this direction. As it happens I was discussing this question with Ben and Matt over dinner last night, and unfortunately they explained to me that it is more difficult than I naively hoped, even once wired-in and known-key things are moved to ghc-internal.

The difficulty is that, as a normal Haskell library, ghc itself will be compiled against a particular version of base. Then when Template Haskell is used (with the internal interpreter), code will be dynamically loaded into a process that already has symbols for ghc's version of base, which means it is not safe for the code to depend on a different version of base. This is rather like the situation with TH and cross-compilers. To avoid that problem, GHC's own dependency on "base" could be indirect via a shared object with versioned symbol names and a version-specific SONAME (possibly even a private to GHC SONAME and private symbol version names). Say "libbase.so.4.19.1".

The problem here is deeper than simply the symbol names. For instance, the `List` type provided by `base-X.Y.Z` and `base-X.Y.Z'` may differ. Since lists are used in the `template-haskell` AST, we would be unable to share lists between `template-haskell` and `ghc`.

As noted in my recent reply elsewhere in the thread, this can be avoided by communicating via serialisation instead of heap objects.

Cheers,

- Ben

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

(Meta-question: on reflection, would this discussion perhaps be better on a ticket? But where? GHC's repo? Or HF's?) The difficulty is that, as a normal Haskell library, ghc itself will be

compiled against a particular verson of base. Then when Template Haskell is used (with the internal interpreter), code will be dynamically loaded into a process that already has symbols for ghc's version of base, which means it is not safe for the code to depend on a different version of base.

I'm not understanding the difficulty yet. Let's say that - An old library mylib (which uses TH) depends on base-4.7. - A new GHC, say GHC 9.10, depends on a newer version of base-4.9, which in turn depends on ghc-internal-9.10. - At the same time, though, we release base-4.7.1, which depends on ghc-internal-9.10, and exposes the base-4.7 API. At this point we use ghc-9.10 to compile L, against base-4.7.1. (Note the the ghc-9.10 binary includes a compiled form of `base-4.9`. - That produces compiled object files, such as, mylib:M.o. - To run TH we need to link them with the running binary - So we need to link the compiled `base-4.7.1` as well. No problem: it contains very little code; it is mostly a shim for ghc-internal-9.10 So the only thing we need is the ability to have a single linked binary that includes (the compiled form for) two different versions/instantiations of `base`. I think that's already supported: each has a distinct "installed package id". What am I missing? Simon On Tue, 17 Oct 2023 at 16:54, Adam Gundry wrote:

Hi Simon,

Thanks for starting this discussion, it would be good to see progress in this direction. As it happens I was discussing this question with Ben and Matt over dinner last night, and unfortunately they explained to me that it is more difficult than I naively hoped, even once wired-in and known-key things are moved to ghc-internal.

The difficulty is that, as a normal Haskell library, ghc itself will be compiled against a particular version of base. Then when Template Haskell is used (with the internal interpreter), code will be dynamically loaded into a process that already has symbols for ghc's version of base, which means it is not safe for the code to depend on a different version of base. This is rather like the situation with TH and cross-compilers.

Adam

On 17/10/2023 11:08, Simon Peyton Jones wrote:

...

Dear GHC devs

Given the now-agreed split between ghc-internal and base https://github.com/haskellfoundation/tech-proposals/pull/51, what stands in the way of a "reinstallable base"?

Specifically, suppose that

* GHC 9.8 comes out with base-4.9 * The CLC decides to make some change to `base`, so we get base-4.10 * Then GHC 9.10 comes out with base-4.10

I think we'd all like it if someone could use GHC 9.10 to compile a library L that depends on base-4.9 and either L doesn't work at all with base-4.10, or L's dependency bounds have not yet been adjusted to allow base-4.10.

We'd like to have a version of `base`, say `base-4.9.1` that has the exact same API as `base-4.9` but works with GHC 9.10.

Today, GHC 9.10 comes with a specific version of base, /and you can't change it/. The original reason for that was, I recall, that GHC knows the precise place where (say) the type Int is declared, and it'll get very confused if that data type definition moves around.

But now we have `ghc-internal`, all these "things that GHC magically knows" are in `ghc-internal`, not `base`.

*Hence my question: what (now) stops us making `base` behave like any other library*? That would be a big step forward, because it would mean that a newer GHC could compile old libraries against their old dependencies.

(Some changes would still be difficult. If, for example, we removed Monad and replaced it with classes Mo1 and Mo2, it might be hard to simulate the old `base` with a shim. But getting 99% of the way there would still be fantastic.)

Simon

-- Adam Gundry, Haskell Consultant Well-Typed LLP, https://www.well-typed.com/

Registered in England & Wales, OC335890 27 Old Gloucester Street, London WC1N 3AX, England

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

iserv doesn't solve issues with GHC plugins (e.g. type-checker plugins)? As far as I understand, if there is a library with a type-checker plugin, then that library is forced to use whatever dependencies `ghc` library is using? I guess that is the fair restriction: such libraries wouldn't be able to use non-bundled `base` On 18.10.2023 5.30, Moritz Angermann wrote:

Something I haven’t gotten around to but only preliminary experiments with is dynamically built iserv binaries.

Using -fexternal-interpreter can decouple the symbols the interpreter sees and those the compiler sees (They can even be of different architectures). iserv could be linked against the base the project wants to use, whereas GHC itself could use a different base. I’m not sure this covers everything, but it covers at least the case where we don’t need to load two different packages into the same process.

Wrt to TH, I’m a bit behind on reading all the prior work to solve this, but conceptually I still believe template-haskell itself should not expose the internal ast, but only a combinator API to it.

Regarding DSO’s: let’s please not make the existence of DSO a hard dependency. There are platforms for which we don’t have DSO capabilities, and where we are forced to use the in-memory loader and linker.

On Wed, 18 Oct 2023 at 4:17 AM, Simon Peyton Jones wrote:

(Meta-question: on reflection, would this discussion perhaps be better on a ticket? But where?  GHC's repo?  Or HF's?)

The difficulty is that, as a normal Haskell library, ghc itself will be compiled against a particular verson of base. Then when Template Haskell is used (with the internal interpreter), code will be dynamically loaded into a process that already has symbols for ghc's version of base, which means it is not safe for the code to depend on a different version of base.

I'm not understanding the difficulty yet.

Let's say that

* An old library mylib (which uses TH) depends on base-4.7. * A new GHC, say GHC 9.10, depends on a newer version of base-4.9, which in turn depends on ghc-internal-9.10. * At the same time, though, we release base-4.7.1, which depends on ghc-internal-9.10, and exposes the base-4.7 API.

At this point we use ghc-9.10 to compile L, against base-4.7.1. (Note the the ghc-9.10 binary includes a compiled form of `base-4.9`.

* That produces compiled object files, such as, mylib:M.o. * To run TH we need to link them with the running binary * So we need to link the compiled `base-4.7.1` as well.  No problem: it contains very little code; it is mostly a shim for ghc-internal-9.10

So the only thing we need is the ability to have a single linked binary that includes (the compiled form for) two different versions/instantiations of `base`.   I think that's already supported: each has a distinct "installed package id".

What am I missing?

Simon

On Tue, 17 Oct 2023 at 16:54, Adam Gundry wrote:

Hi Simon,

Thanks for starting this discussion, it would be good to see progress in this direction. As it happens I was discussing this question with Ben and Matt over dinner last night, and unfortunately they explained to me that it is more difficult than I naively hoped, even once wired-in and known-key things are moved to ghc-internal.

The difficulty is that, as a normal Haskell library, ghc itself will be compiled against a particular version of base. Then when Template Haskell is used (with the internal interpreter), code will be dynamically loaded into a process that already has symbols for ghc's version of base, which means it is not safe for the code to depend on a different version of base. This is rather like the situation with TH and cross-compilers.

Adam

On 17/10/2023 11:08, Simon Peyton Jones wrote: > Dear GHC devs > > Given the now-agreed split between ghc-internal and base > https://github.com/haskellfoundation/tech-proposals/pull/51, what > stands in the way of a "reinstallable base"? > > Specifically, suppose that > >   * GHC 9.8 comes out with base-4.9 >   * The CLC decides to make some change to `base`, so we get base-4.10 >   * Then GHC 9.10 comes out with base-4.10 > > I think we'd all like it if someone could use GHC 9.10 to compile a > library L that depends on base-4.9 and either L doesn't work at all with > base-4.10, or L's dependency bounds have not yet been adjusted to allow > base-4.10. > > We'd like to have a version of `base`, say `base-4.9.1` that has the > exact same API as `base-4.9` but works with GHC 9.10. > > Today, GHC 9.10 comes with a specific version of base, /and you can't > change it/. The original reason for that was, I recall, that GHC knows > the precise place where (say) the type Int is declared, and it'll get > very confused if that data type definition moves around. > > But now we have `ghc-internal`, all these "things that GHC magically > knows" are in `ghc-internal`, not `base`. > > *Hence my question: what (now) stops us making `base` behave like any > other library*?  That would be a big step forward, because it would mean > that a newer GHC could compile old libraries against their old dependencies. > > (Some changes would still be difficult.  If, for example, we removed > Monad and replaced it with classes Mo1 and Mo2, it might be hard to > simulate the old `base` with a shim.  But getting 99% of the way there > would still be fantastic.) > > Simon

-- Adam Gundry, Haskell Consultant Well-Typed LLP, https://www.well-typed.com/

Registered in England & Wales, OC335890 27 Old Gloucester Street, London WC1N 3AX, England

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

For what it worth, `template-haskell` itself depends on a `base`. So if `base` if different base is used, different `template-haskell` is to be used. In my opinion is not *too unfair* to require that if you actually splice in (i.e. the code not only provides template-haskell combinators to create/modify splices) then you must have base and template-haskell versions aligned with host GHC used versions. The same restriction is GHC plugins, isn't it, except `template-haskell` is replaced with `ghc`? - Oleg On 17.10.2023 18.54, Adam Gundry wrote:

Hi Simon,

Thanks for starting this discussion, it would be good to see progress in this direction. As it happens I was discussing this question with Ben and Matt over dinner last night, and unfortunately they explained to me that it is more difficult than I naively hoped, even once wired-in and known-key things are moved to ghc-internal.

The difficulty is that, as a normal Haskell library, ghc itself will be compiled against a particular version of base. Then when Template Haskell is used (with the internal interpreter), code will be dynamically loaded into a process that already has symbols for ghc's version of base, which means it is not safe for the code to depend on a different version of base. This is rather like the situation with TH and cross-compilers.

Adam

On 17/10/2023 11:08, Simon Peyton Jones wrote:

...

Dear GHC devs

Given the now-agreed split between ghc-internal and base https://github.com/haskellfoundation/tech-proposals/pull/51, what stands in the way of a "reinstallable base"?

Specifically, suppose that

  * GHC 9.8 comes out with base-4.9   * The CLC decides to make some change to `base`, so we get base-4.10   * Then GHC 9.10 comes out with base-4.10

I think we'd all like it if someone could use GHC 9.10 to compile a library L that depends on base-4.9 and either L doesn't work at all with base-4.10, or L's dependency bounds have not yet been adjusted to allow base-4.10.

We'd like to have a version of `base`, say `base-4.9.1` that has the exact same API as `base-4.9` but works with GHC 9.10.

Today, GHC 9.10 comes with a specific version of base, /and you can't change it/. The original reason for that was, I recall, that GHC knows the precise place where (say) the type Int is declared, and it'll get very confused if that data type definition moves around.

But now we have `ghc-internal`, all these "things that GHC magically knows" are in `ghc-internal`, not `base`.

*Hence my question: what (now) stops us making `base` behave like any other library*?  That would be a big step forward, because it would mean that a newer GHC could compile old libraries against their old dependencies.

(Some changes would still be difficult.  If, for example, we removed Monad and replaced it with classes Mo1 and Mo2, it might be hard to simulate the old `base` with a shim.  But getting 99% of the way there would still be fantastic.)

Simon

A very large proportion of libraries, and virtually all end-user applications, transitively depend on Template Haskell. Whether they use Template Haskell directly or not. So if we're saying “base is reinstallable, except when you have Template Haskell somewhere”, we're effectively saying “base is not reinstallable”. Now, it could be a good stepping-stone, from an engineering standpoint, but I don't think we could deliver this and be satisfied that we've accomplished anything.

No one has yet answered my naive question (from 3 days ago) asking why Template Haskell stops base being reinstallable. I'll quote it here for completeness. Let's say that

- An old library mylib (which uses TH) depends on base-4.7. - A new GHC, say GHC 9.10, depends on a newer version of base-4.9, which in turn depends on ghc-internal-9.10. - At the same time, though, we release base-4.7.1, which depends on ghc-internal-9.10, and exposes the base-4.7 API. At this point we use ghc-9.10 to compile L, against base-4.7.1. (Note that the ghc-9.10 binary includes a compiled form of `base-4.9`.) - That produces compiled object files, such as, mylib:M.o. - To run TH we need to link them with the running binary - So we need to link the compiled `base-4.7.1` as well. No problem: it contains very little code; it is mostly a shim for ghc-internal-9.10 So the only thing we need is the ability to have a single linked binary that includes (the compiled form for) two different versions/instantiations of `base`. I think that's already supported: each has a distinct "installed package id". (End of quote) What am I missing? Simon On Fri, 20 Oct 2023 at 08:57, Arnaud Spiwack wrote:

A very large proportion of libraries, and virtually all end-user applications, transitively depend on Template Haskell. Whether they use Template Haskell directly or not. So if we're saying “base is reinstallable, except when you have Template Haskell somewhere”, we're effectively saying “base is not reinstallable”. Now, it could be a good stepping-stone, from an engineering standpoint, but I don't think we could deliver this and be satisfied that we've accomplished anything.

On Thu, 19 Oct 2023 at 13:47, Oleg Grenrus wrote:

...

For what it worth, `template-haskell` itself depends on a `base`. So if `base` if different base is used, different `template-haskell` is to be used.

In my opinion is not *too unfair* to require that if you actually splice in (i.e. the code not only provides template-haskell combinators to create/modify splices) then you must have base and template-haskell versions aligned with host GHC used versions.

The same restriction is GHC plugins, isn't it, except `template-haskell` is replaced with `ghc`?

- Oleg

On 17.10.2023 18.54, Adam Gundry wrote:

...

Hi Simon,

Thanks for starting this discussion, it would be good to see progress in this direction. As it happens I was discussing this question with Ben and Matt over dinner last night, and unfortunately they explained to me that it is more difficult than I naively hoped, even once wired-in and known-key things are moved to ghc-internal.

The difficulty is that, as a normal Haskell library, ghc itself will be compiled against a particular version of base. Then when Template Haskell is used (with the internal interpreter), code will be dynamically loaded into a process that already has symbols for ghc's version of base, which means it is not safe for the code to depend on a different version of base. This is rather like the situation with TH and cross-compilers.

Adam

On 17/10/2023 11:08, Simon Peyton Jones wrote:

...

Dear GHC devs

Given the now-agreed split between ghc-internal and base https://github.com/haskellfoundation/tech-proposals/pull/51, what stands in the way of a "reinstallable base"?

Specifically, suppose that

* GHC 9.8 comes out with base-4.9 * The CLC decides to make some change to `base`, so we get base-4.10 * Then GHC 9.10 comes out with base-4.10

I think we'd all like it if someone could use GHC 9.10 to compile a library L that depends on base-4.9 and either L doesn't work at all with base-4.10, or L's dependency bounds have not yet been adjusted to allow base-4.10.

We'd like to have a version of `base`, say `base-4.9.1` that has the exact same API as `base-4.9` but works with GHC 9.10.

Today, GHC 9.10 comes with a specific version of base, /and you can't change it/. The original reason for that was, I recall, that GHC knows the precise place where (say) the type Int is declared, and it'll get very confused if that data type definition moves around.

But now we have `ghc-internal`, all these "things that GHC magically knows" are in `ghc-internal`, not `base`.

*Hence my question: what (now) stops us making `base` behave like any other library*? That would be a big step forward, because it would mean that a newer GHC could compile old libraries against their old dependencies.

(Some changes would still be difficult. If, for example, we removed Monad and replaced it with classes Mo1 and Mo2, it might be hard to simulate the old `base` with a shim. But getting 99% of the way there would still be fantastic.)

Simon

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- Arnaud Spiwack Director, Research at https://moduscreate.com and https://tweag.io. _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Hi Simon, As this came up again during the GHC call yesterday, I've written some notes to explain the issues and the current status (as I understand them): https://hsyl20.fr/posts/2024-08-07-about-ghcs-stability.html I hope that helps. Happy to get some feedback about this by other people involved in the process. Sylvain On 20/10/2023 10:00, Simon Peyton Jones wrote:

A very large proportion of libraries, and virtually all end-user applications, transitively depend on Template Haskell. Whether they use Template Haskell directly or not. So if we're saying “base is reinstallable, except when you have Template Haskell somewhere”, we're effectively saying “base is not reinstallable”. Now, it could be a good stepping-stone, from an engineering standpoint, but I don't think we could deliver this and be satisfied that we've accomplished anything.

No one has yet answered my naive question (from 3 days ago) asking why Template Haskell stops base being reinstallable. I'll quote it here for completeness.

Let's say that

* An old library mylib (which uses TH) depends on base-4.7. * A new GHC, say GHC 9.10, depends on a newer version of base-4.9, which in turn depends on ghc-internal-9.10. * At the same time, though, we release base-4.7.1, which depends on ghc-internal-9.10, and exposes the base-4.7 API.

At this point we use ghc-9.10 to compile L, against base-4.7.1.   (Note that the ghc-9.10 binary includes a compiled form of `base-4.9`.)

* That produces compiled object files, such as, mylib:M.o. * To run TH we need to link them with the running binary * So we need to link the compiled `base-4.7.1` as well.  No problem: it contains very little code; it is mostly a shim for ghc-internal-9.10

So the only thing we need is the ability to have a single linked binary that includes (the compiled form for) two different versions/instantiations of `base`.   I think that's already supported: each has a distinct "installed package id".

(End of quote)

What am I missing?

Simon

On Fri, 20 Oct 2023 at 08:57, Arnaud Spiwack wrote:

A very large proportion of libraries, and virtually all end-user applications, transitively depend on Template Haskell. Whether they use Template Haskell directly or not. So if we're saying “base is reinstallable, except when you have Template Haskell somewhere”, we're effectively saying “base is not reinstallable”. Now, it could be a good stepping-stone, from an engineering standpoint, but I don't think we could deliver this and be satisfied that we've accomplished anything.

On Thu, 19 Oct 2023 at 13:47, Oleg Grenrus wrote:

For what it worth, `template-haskell` itself depends on a `base`. So if `base` if different base is used, different `template-haskell` is to be used.

In my opinion is not *too unfair* to require that if you actually splice in (i.e. the code not only provides template-haskell combinators to create/modify splices) then you must have base and template-haskell versions aligned with host GHC used versions.

The same restriction is GHC plugins, isn't it, except `template-haskell` is replaced with `ghc`?

- Oleg

On 17.10.2023 18.54, Adam Gundry wrote: > Hi Simon, > > Thanks for starting this discussion, it would be good to see progress > in this direction. As it happens I was discussing this question with > Ben and Matt over dinner last night, and unfortunately they explained > to me that it is more difficult than I naively hoped, even once > wired-in and known-key things are moved to ghc-internal. > > The difficulty is that, as a normal Haskell library, ghc itself will > be compiled against a particular version of base. Then when Template > Haskell is used (with the internal interpreter), code will be > dynamically loaded into a process that already has symbols for ghc's > version of base, which means it is not safe for the code to depend on > a different version of base. This is rather like the situation with TH > and cross-compilers. > > Adam > > > > On 17/10/2023 11:08, Simon Peyton Jones wrote: >> Dear GHC devs >> >> Given the now-agreed split between ghc-internal and base >> https://github.com/haskellfoundation/tech-proposals/pull/51, what >> stands in the way of a "reinstallable base"? >> >> Specifically, suppose that >> >>   * GHC 9.8 comes out with base-4.9 >>   * The CLC decides to make some change to `base`, so we get base-4.10 >>   * Then GHC 9.10 comes out with base-4.10 >> >> I think we'd all like it if someone could use GHC 9.10 to compile a >> library L that depends on base-4.9 and either L doesn't work at all >> with base-4.10, or L's dependency bounds have not yet been adjusted >> to allow base-4.10. >> >> We'd like to have a version of `base`, say `base-4.9.1` that has the >> exact same API as `base-4.9` but works with GHC 9.10. >> >> Today, GHC 9.10 comes with a specific version of base, /and you can't >> change it/. The original reason for that was, I recall, that GHC >> knows the precise place where (say) the type Int is declared, and >> it'll get very confused if that data type definition moves around. >> >> But now we have `ghc-internal`, all these "things that GHC magically >> knows" are in `ghc-internal`, not `base`. >> >> *Hence my question: what (now) stops us making `base` behave like any >> other library*?  That would be a big step forward, because it would >> mean that a newer GHC could compile old libraries against their old >> dependencies. >> >> (Some changes would still be difficult.  If, for example, we removed >> Monad and replaced it with classes Mo1 and Mo2, it might be hard to >> simulate the old `base` with a shim.  But getting 99% of the way >> there would still be fantastic.) >> >> Simon > _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- Arnaud Spiwack Director, Research at https://moduscreate.com and https://tweag.io. _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

I think it is making progress. It's not just an engineering milestone but also messaging. Right now a huge issue that every Haskeller faces is new GHCs having new major versions of base. If we make clear that "it's not base's fault, it's Template Haskell's fault", then we will have an easier time coordinating people and eventually fundraising to get issued fixed with Template Haskell. John On 10/20/23 03:56, Arnaud Spiwack wrote:

A very large proportion of libraries, and virtually all end-user applications, transitively depend on Template Haskell. Whether they use Template Haskell directly or not. So if we're saying “base is reinstallable, except when you have Template Haskell somewhere”, we're effectively saying “base is not reinstallable”. Now, it could be a good stepping-stone, from an engineering standpoint, but I don't think we could deliver this and be satisfied that we've accomplished anything.

On Thu, 19 Oct 2023 at 13:47, Oleg Grenrus wrote:

For what it worth, `template-haskell` itself depends on a `base`. So if `base` if different base is used, different `template-haskell` is to be used.

In my opinion is not *too unfair* to require that if you actually splice in (i.e. the code not only provides template-haskell combinators to create/modify splices) then you must have base and template-haskell versions aligned with host GHC used versions.

The same restriction is GHC plugins, isn't it, except `template-haskell` is replaced with `ghc`?

- Oleg

On 17.10.2023 18.54, Adam Gundry wrote: > Hi Simon, > > Thanks for starting this discussion, it would be good to see progress > in this direction. As it happens I was discussing this question with > Ben and Matt over dinner last night, and unfortunately they explained > to me that it is more difficult than I naively hoped, even once > wired-in and known-key things are moved to ghc-internal. > > The difficulty is that, as a normal Haskell library, ghc itself will > be compiled against a particular version of base. Then when Template > Haskell is used (with the internal interpreter), code will be > dynamically loaded into a process that already has symbols for ghc's > version of base, which means it is not safe for the code to depend on > a different version of base. This is rather like the situation with TH > and cross-compilers. > > Adam > > > > On 17/10/2023 11:08, Simon Peyton Jones wrote: >> Dear GHC devs >> >> Given the now-agreed split between ghc-internal and base >> https://github.com/haskellfoundation/tech-proposals/pull/51, what >> stands in the way of a "reinstallable base"? >> >> Specifically, suppose that >> >>   * GHC 9.8 comes out with base-4.9 >>   * The CLC decides to make some change to `base`, so we get base-4.10 >>   * Then GHC 9.10 comes out with base-4.10 >> >> I think we'd all like it if someone could use GHC 9.10 to compile a >> library L that depends on base-4.9 and either L doesn't work at all >> with base-4.10, or L's dependency bounds have not yet been adjusted >> to allow base-4.10. >> >> We'd like to have a version of `base`, say `base-4.9.1` that has the >> exact same API as `base-4.9` but works with GHC 9.10. >> >> Today, GHC 9.10 comes with a specific version of base, /and you can't >> change it/. The original reason for that was, I recall, that GHC >> knows the precise place where (say) the type Int is declared, and >> it'll get very confused if that data type definition moves around. >> >> But now we have `ghc-internal`, all these "things that GHC magically >> knows" are in `ghc-internal`, not `base`. >> >> *Hence my question: what (now) stops us making `base` behave like any >> other library*?  That would be a big step forward, because it would >> mean that a newer GHC could compile old libraries against their old >> dependencies. >> >> (Some changes would still be difficult.  If, for example, we removed >> Monad and replaced it with classes Mo1 and Mo2, it might be hard to >> simulate the old `base` with a shim.  But getting 99% of the way >> there would still be fantastic.) >> >> Simon > _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-- Arnaud Spiwack Director, Research at https://moduscreate.com and https://tweag.io.

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Ben Gamari writes:

Simon Peyton Jones writes:

...

Dear GHC devs

Given the now-agreed split between ghc-internal and base https://github.com/haskellfoundation/tech-proposals/pull/51, what stands in the way of a "reinstallable base"?

Specifically, suppose that

- GHC 9.8 comes out with base-4.9 - The CLC decides to make some change to `base`, so we get base-4.10 - Then GHC 9.10 comes out with base-4.10

We thought about this quite a bit at the Well-Typed meeting this week. In short, I suspect that after the `ghc-internal` split is merged and some further work we will be able to use multiple `base` versions with a single compiler version.

I imagine this would be done in roughly three phases. The first two phases can happen in the short term; the last is a bit further off. I'll describe these in turn below.

One final point that I forgot to mention: None of this addresses the problem of compiler plugins. This is problematic as `Cabal`'s one-version-per-install-plan restriction means means that any package using a plugin will be forced to use the precise `base` that `ghc` itself is linked against. I can think of three ways to address this: * Teach `Cabal` to distinguish between plugin dependencies (which are only needed at build time and needn't be linked into final build results) and normal runtime dependencies. This is not a complete solution as many plugins have both compile-time and runtime components, but it would help in some cases. * Make `ghc` reinstallable. This allows `Cabal` to rebuild the compiler when we need to link against a different `base`. We have started this process but in general it is quite tricky to get right (see #20742) and will require cooperation from `Cabal` . * Serialise structures used by plugins, just as we do for the TH AST. Unfortunately, the surface area of the plugin interface is significantly larger than that of TH. None of these options are easy. For this reason, I think it would be wise to leave plugins as future work (if we decide to address it at all).

Ben Gamari writes:

...

# Phase 2: Template Haskell

The above story is made considerably messier by `template-haskell`. ...

One further clarification as I fear this may be unclear: Phase 2 does *not* allow multiple versions of `template-haskell` to be used with a single GHC. That is, each GHC version will have precisely one corresponding `template-haskell` version; consequently, programs using TemplateHaskell's combinator's will still need to adapt to new `template-haskell` releases when moving to a new GHC. Phase 3 describes how we could eliminate this need by allowing multiple `template-haskell` versions to be used with a single GHC release.