In principle, I think we should treat the GHC API like any other library, and try not to break code unnecessarily. However, my impression is that the GHC API grew somewhat organically, so we may want to put some additional work before we stabilize things too much. It's been a while since I used it, so I might be out of date, but last I looked the GHC API was a module exporting some high-level functions from GHC. I think that a single module is too small of an API for a project as large as GHC. In fact, it probably makes sense to define more than one API. For example, each plugin point should probably have its own API, and that's likely different to the GHC API that exposes functionality such as "load and type check this module here", or "parse and evaluate this string". -Iavor On Mon, Jul 27, 2020 at 4:05 AM Simon Peyton Jones via ghc-devs < ghc-devs@haskell.org> wrote:

What I’m after is a clear opportunity for informed debate, and a clear yes/no decision. That need not be high overhead.

It means paying some upfront cost for design changes. But that’s better than the legacy cost of dealing with things that turn out, in retrospect, to be less well designed than they could be.

We tend to think of APIs as implementation details. But they are deeply significant, and express key abstractions, just like language designs do. I think we should treat them just as seriously.

Simon

*From:* Mathieu Boespflug *Sent:* 27 July 2020 11:11 *To:* Simon Peyton Jones *Cc:* ghc-devs@haskell.org Devs *Subject:* Re: How should we treat changes to the GHC API?

I would just point out that decision by committee, and in particular the GHC Proposals process, has a high cost in terms of both total human brain cycles and latency. The cost is entirely justified when it comes to things that are a) hard to revert and b) extremely hard to get right the first time, like new extensions to the language, or c) very sensitive (like governance, say). For things like breaking changes to API's, it's worth writing out what the current problems are. Are users complaining that the API churn is too high? Are they concerned about endemic quality problems with the API?

It may be enough to make sure to know who the main users of the API are and tag them as reviewers on these types of changes in GitLab. Or to avoid extra process but enshrine principles that might be necessary to adopt, like saying that existing API functions should always be kept as-is during some deprecation period and new functionality should be exposed in new additions to the API. Principles to be upheld by reviewers.

On Mon, Jul 27, 2020 at 10:45:50, Simon Peyton Jones wrote:

A recent MR for GHC https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2F-%2Fmerge_requests%2F3758&data=02%7C01%7Csimonpj%40microsoft.com%7C730e52088cb64dcebe1408d8321567e1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637314414803966186&sdata=GCGrNinXtNxPcJfEu%2F%2BifrJJ22BB2bkIy4E9d5IWOuo%3D&reserved=0 (adding machinery for plugins to write data to extensible interface files) made me wonder:

how we should treat significant changes to the GHC API?

Changes to the GHC API, especially to bits used by plugins or by IDEs, are clearly user-visible to an important class of users – they are not just internal to GHC itself. So, how should we review them? Should they perhaps be part of the GHC proposals process? Or some other similar process? (The collection of experts on the GHC API, plugins, IDEs etc, is rather different to the membership of the GHC steering group.)

I'm asking, not to be obstructive, but because the GHC API deserves to be thought of as a whole; in the past it has grown incrementally, without much discussion, and that has not served us well. But at the moment there is no process, no group to consult.

Any views?

Simon

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haskell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-devs&data=02%7C01%7Csimonpj%40microsoft.com%7C730e52088cb64dcebe1408d8321567e1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637314414803966186&sdata=2TwuVzxKm88aevbTooLG3sLeakrSYZziFPNDozFCvHo%3D&reserved=0

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

I would say the current API is....absolutely terrible and so we should have no qualms about breaking it. Talking to the people who work on the IDE---i.e. the people who are the most adversely impacted by churn---they also don't like it and not want it stabilized in its current form. I think the only good long term solution is to modularize GHC into multiple packages, giving us conceptually distinct interfaces which break at different rates. This is similar to modularizing base as e.g. GHC.* breaks far more often than e.g. Data.List. How we evaluate changes is less important to me as my main priority is to get the pace of development much higher so we have a chance on fixing all the technical debt to get us out of the current situation. Basically, you could count me in "team heat bath"; I think that should have the right connotations. John On 7/27/20 2:07 PM, Eric Seidel wrote:

As another former user of the GHC API, I'd say my two biggest complaints were the relative instability of the API and the lack of documentation. I haven't used the API in at least three years though, so it's possible much has changed since my experience.

I remember often having to do significant work to adapt LiquidHaskell to new versions of GHC due to small changes in the API. We often felt like we had to import internal modules (ie other than 'GHC') to get key bits of functionality we needed, which might explain the churn. But it also points to Iavor's point that the public API grew organically and might benefit from a bit of top-down design to make sure it's complete enough for typical use cases.

For documentation, the issue was less the API docs but a lack of "How do I do X?" docs and examples. One problem that I remember being particularly vexing was resolving names in a particular scope (in my case it was always module-level scopes, but I can easily imagine clients that would want to resolve names in local scopes).

I don't know if the API needs to go through something like the Steering Committee, but a stronger focus on API stability and perhaps a broader view of what constitutes (or should be included in) the public-facing API would be welcome!

On Mon, Jul 27, 2020, at 11:45, Iavor Diatchki wrote:

...

In principle, I think we should treat the GHC API like any other library, and try not to break code unnecessarily. However, my impression is that the GHC API grew somewhat organically, so we may want to put some additional work before we stabilize things too much. It's been a while since I used it, so I might be out of date, but last I looked the GHC API was a module exporting some high-level functions from GHC. I think that a single module is too small of an API for a project as large as GHC. In fact, it probably makes sense to define more than one API. For example, each plugin point should probably have its own API, and that's likely different to the GHC API that exposes functionality such as "load and type check this module here", or "parse and evaluate this string".

-Iavor

On Mon, Jul 27, 2020 at 4:05 AM Simon Peyton Jones via ghc-devs wrote:

...

What I’m after is a clear opportunity for informed debate, and a clear yes/no decision. That need not be high overhead.____ __ __ It means paying some upfront cost for design changes. But that’s better than the legacy cost of dealing with things that turn out, in retrospect, to be less well designed than they could be.____ __ __ We tend to think of APIs as implementation details. But they are deeply significant, and express key abstractions, just like language designs do. I think we should treat them just as seriously.____ __ __ Simon____ __ __ *From:* Mathieu Boespflug *Sent:* 27 July 2020 11:11 *To:* Simon Peyton Jones *Cc:* ghc-devs@haskell.org Devs *Subject:* Re: How should we treat changes to the GHC API?____ __ __ I would just point out that decision by committee, and in particular the GHC Proposals process, has a high cost in terms of both total human brain cycles and latency. The cost is entirely justified when it comes to things that are a) hard to revert and b) extremely hard to get right the first time, like new extensions to the language, or c) very sensitive (like governance, say). For things like breaking changes to API's, it's worth writing out what the current problems are. Are users complaining that the API churn is too high? Are they concerned about endemic quality problems with the API?____ __ __ It may be enough to make sure to know who the main users of the API are and tag them as reviewers on these types of changes in GitLab. Or to avoid extra process but enshrine principles that might be necessary to adopt, like saying that existing API functions should always be kept as-is during some deprecation period and new functionality should be exposed in new additions to the API. Principles to be upheld by reviewers.____ __ __ On Mon, Jul 27, 2020 at 10:45:50, Simon Peyton Jones wrote:____

...

A recent MR for GHC https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2F-%2Fmerge_requests%2F3758&data=02%7C01%7Csimonpj%40microsoft.com%7C730e52088cb64dcebe1408d8321567e1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637314414803966186&sdata=GCGrNinXtNxPcJfEu%2F%2BifrJJ22BB2bkIy4E9d5IWOuo%3D&reserved=0 (adding machinery for plugins to write data to extensible interface files) made me wonder: ____ how we should treat significant changes to the GHC API?____ Changes to the GHC API, especially to bits used by plugins or by IDEs, are clearly user-visible to an important class of users – they are not just internal to GHC itself. So, how should we review them? Should they perhaps be part of the GHC proposals process? Or some other similar process? (The collection of experts on the GHC API, plugins, IDEs etc, is rather different to the membership of the GHC steering group.)____ I'm asking, not to be obstructive, but because the GHC API deserves to be thought of as a whole; in the past it has grown incrementally, without much discussion, and that has not served us well. But at the moment there is no process, no group to consult.____ Any views?____ Simon____ _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haskell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-devs&data=02%7C01%7Csimonpj%40microsoft.com%7C730e52088cb64dcebe1408d8321567e1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637314414803966186&sdata=2TwuVzxKm88aevbTooLG3sLeakrSYZziFPNDozFCvHo%3D&reserved=0____

---

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

---

ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

On 7/27/20 4:57 PM, Richard Eisenberg wrote

On the other hand, we could imagine a dedicated GHC API volunteer who maintains the API on top of the shifting sands of GHC. Having a central project to deal with the API would satisfy clients by projecting a stable interface, and it wouldn't hinder GHC development, as the API volunteer would take on the compatibility burden. Of course, success here would require a dedicated, competent volunteer.

I think that basically exists: https://hackage.haskell.org/package/ghc-lib ? As permanent solution I don't like it, but as a stop gap to relieve any pressure for stability until we have sane interfaces, I *love* it. John

On Mon, Jul 27, 2020 at 7:13 PM Shayne Fletcher < shayne.fletcher.50@gmail.com> wrote:

On Mon, Jul 27, 2020 at 5:24 PM John Ericson wrote:

...

I think that basically exists: https://hackage.haskell.org/package/ghc-lib ? As permanent solution I don't like it, but as a stop gap to relieve any pressure for stability until we have sane interfaces, I *love* it.

`ghc-lib` is a literal subset of GHC files: `ghc-lib-parser` that set of files sufficient to produce abstract syntax trees (~200 files), `ghc-lib` the remaining set of files enabling Core generation from parsed syntax trees (~300 of those) (more detail in the project README here https://github.com/digital-asset/ghc-lib/blob/master/README.md).

There is at this time, no attempt to provide any additional interface, "higher level" or otherwise, in these packages.

[...]

I should mention that the related project `ghc-lib-parser-ex` (README here https://github.com/shayne-fletcher/ghc-lib-parser-ex/blob/master/README.md) IS in large part motivated by the need to provide stable interfaces over `ghc-lib-parser`. -- Shayne Fletcher

On 7/27/20 10:57 PM, Richard Eisenberg wrote:

I guess I'm arguing that ghc-lib (or something like it) should be the permanent solution. GHC will always be in flux internally.

So to be clear I don't disagree that GHC will always be in flux. But I think things like ghc-lib largely work today because GHC is so imperative. Once it (hopefully!) is more functional, we'll have the problem less that functions change[1], and more that data structures change. I think that will be harder to abstract over---often to migrate an interface properly, data structure will need migrations in both directions *and* that those migrations form an isomorphism, which can be an impossible requirement. It's just a fact of life that rich data structures make migrations harder, but that's no reason we shouldn't have rich data structures! I'm probably too deeply exploring what's only one possible future at this point, so let me rewind out of that exact hypothesis and just make the broader point that if the interfaces change a lot, a lot of organizational stuff can/should too. If the interface changes *don't* engender cultural shifts around working with GHC, we're probably doing them wrong and they are vapid churn after all! John [1]: there should be way fewer functions than today. we currently have a gratuitous combinatorial explosion of little helpers which obscures what the underlying orthogonal concepts are.

On Mon, Jul 27, 2020 at 08:57:12PM +0000, Richard Eisenberg wrote:

On the other hand, we could imagine a dedicated GHC API volunteer who maintains the API on top of the shifting sands of GHC.

Looking at other compilers that have been successful in having a stable API - like clang with the libclang - that's pretty much how they achieved it. The manpower of GHC devs is already pretty small, so putting another burden on them with a stable API won't work out that well. Also as a compiler dev you want to have the freedom to change your AST in the ways you need it, to incorporate new features. Having workarounds at this level to ensure API stability just seems to be the wrong place and will only increase the complexity in an already quite complex project. However a high level API on top of the AST is the perfect place for such special cases between different compiler versions. Greetings, Daniel

My real question is this: * Who “owns” (as in: takes responsibility for) the GHC API? If no person, or no group of people, feels responsible for it, it’ll just grow in a disorganised fashion, which is why we are where we are. This discussion thread has focused mainly on stability, but there is more to it than that. For example: * What kind of stability strikes a reasonable balance between developers and clients? Personally I think we should have * A public API that is pretty stable, plus * Access to many internal GHC functions that change quite a bit. Heavily used internal functions can get “blessed” and moved to the public API. Someone needs to do the blessing. * What is a good public API? Where is it written down? * What should GHC’s extensibility interface be like? Plugins and all that. What is a good design for (say) extensible interface files? What “hooks” should the GHC API afford? This is more than just “what arguments should this function take”… it’s a matter of fundamental design. But design questions like this belong in the GHC-API world (not the core GHC world) because they are all about extension points. * What is a good design for the plugins story more generally? I’d be quite content with a fairly informal group to act as the “owner”. But I think we would be better served if we had such a group, and knew who they were. Simon From: ghc-devs On Behalf Of Neil Mitchell Sent: 28 July 2020 09:19 To: GHC developers Subject: Re: How should we treat changes to the GHC API? The API changing regularly is a real hassle for people building on top of it. But, the people building on top of it are also some of the people changing it most. And the API isn't nice and beautiful as it stands, with lots of horrible work-arounds for downstream users. I'd suggest letting people change the API freely. But avoid making bikeshed API changes (those which are minor improvements at best, and really just a different flavour of the same). And perhaps loop in heavy API users when removing functionality. Thanks, Neil On Tue, Jul 28, 2020 at 7:38 AM Daniel Trstenjak mailto:daniel.trstenjak@gmail.com> wrote: On Mon, Jul 27, 2020 at 08:57:12PM +0000, Richard Eisenberg wrote:

On the other hand, we could imagine a dedicated GHC API volunteer who maintains the API on top of the shifting sands of GHC.

Looking at other compilers that have been successful in having a stable API - like clang with the libclang - that's pretty much how they achieved it. The manpower of GHC devs is already pretty small, so putting another burden on them with a stable API won't work out that well. Also as a compiler dev you want to have the freedom to change your AST in the ways you need it, to incorporate new features. Having workarounds at this level to ensure API stability just seems to be the wrong place and will only increase the complexity in an already quite complex project. However a high level API on top of the AST is the perfect place for such special cases between different compiler versions. Greetings, Daniel _______________________________________________ ghc-devs mailing list ghc-devs@haskell.orgmailto:ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devshttps://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haskell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-devs&data=02%7C01%7Csimonpj%40microsoft.com%7C33d57a82355d47cff09b08d832ceec96%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637315211602862361&sdata=tzH8F0AkBGHuVc%2F1YYkzWV7Lxe0i%2FMWHoJw4Y8olBH8%3D&reserved=0

On 7/31/20 9:27 AM, Simon Peyton Jones via ghc-devs wrote:

My real question is this:

* Who “owns” (as in: takes responsibility for) the GHC API?

I guess one thing very important to me is that the architecture of GHC and it's public interfaces will always be deeply intertwined; or put a different way, efforts to manager the public interface as a thin veneer over a big black box will not work.

* What *is* a good public API?  Where is it written down?

This I agree is important to discuss. I do agree with Moritz that a lot of this stuff can be evolved, but the general direction can be still be discussed. For example, the basic objects I offered are a *huge* departure from what we have today, and any huge change should be discussed up front a bit.

* What should GHC’s extensibility interface be like?   Plugins and all that.  What is a good design for (say) extensible interface files?  What “hooks” should the GHC API afford? This is more than just “what arguments should this function take”… it’s a matter of fundamental design.   But design questions like this belong in the GHC-API world (not the core GHC world) because they are all about extension points. * What is a good design for the plugins story more generally?

I've mentioned this before but I think plugins/hooks/etc. are a terrible way to reuse GHC for other purposes, and exist as a symptom of the rest of the compiler not being at all modular. It's fine that we continue to support them in the short term, but in the long term I'd really like to have a plan to obviate them completely. (One can search "composition over configuration" and variations on that slogan to find much ink has been spilled on this general principle.) John