Hi Gershom, thanks for the extra input. So we've confirmed two facts: * GHC (intended to) ship with only Cabal-2.0 support, but there was a mistake in the release notes so this was unclear to downstream tooling authors. * Cabal-2.0 was released anywhere between slightly *after* and *exactly at the same as* GHC, despite GHC itself shipping with Cabal-2.0. I'm not too concerned by the first point: so long as Cabal-X does not introduce breaking changes, the fact that GHC-Y ultimately shipped with Cabal-X shouldn't be a problem. And this kind of bug in the release notes should go away provided more automation. The second one is more interesting. It is, as you point out, a product of GHC and Cabal being intimately linked and co-developed to a large extent. This leads to a simultaneous release that poses a concrete problem: * if new Cabal versions are used immediately in GHC, then that gives no time at all ahead of a GHC release for downstream tooling authors to adapt, because Cabal is, up until the point of the GHC release, a moving target. Three possible solutions: * Provided no API breaking changes in Cabal, if no metadata that ships with GHC uses new Cabal features for some period of time before release, then the problem goes away. * Or something close to what Manuel proposed in another thread: ship in GHC-X+1 the Cabal version that was co-developed during the development cycle of GHC-X. * Or a middle ground: make feature freeze a thing. Meaning that for a couple of months before a major GHC release, the major new Cabal isn't technically released yet, but like GHC itself within this period, it's pretty staid, so not so much a moving target, and something downstream tooling authors can possibly adapt to even without any grace period on new metadata features. This assumes that the 2 months of feature freeze are enough time for downstream tooling. Thoughts from any of those maintainers? On 14 December 2017 at 01:27, Gershom B wrote:

On Wed, Dec 13, 2017 at 7:06 PM, Boespflug, Mathieu wrote:

...

But crucially, what *is* the policy around Cabal versions? This comment, https://ghc.haskell.org/trac/ghc/ticket/14558#comment:23 claims "if Stack doesn't support the version of Cabal that ships with a certain version of GHC, it shouldn't claim that it supports that version of GHC. The same applies to cabal-install". Is any build tool linked against Cabal-X by definition "not a supported configuration" by GHC-Z if it ships with Cabal-Y such that X < Y?

My understanding is that this is the general thought, yes. In fact, I've been told that even though cabal-install 1.24 did end up working with the GHC 8.2.x series, the release notes, which were not updated properly, actually _were supposed_ to say cabal-install 2.0.0.0 was what was supported there. I believe future cabal-installs will warn when used with a ghc with a newer Cabal-lib than they were built against...

...

Right. But switching from Cabal-2 to Cabal-3 (a hypothetical at this point) sounds like a whole new set of features transitively just made it into the compiler. Is that something we're happy to happen during feature freeze?

Right. After freeze, the compiler itself shouldn't switch from Cabal-2 to Cabal-3. But I would imagine rather that the Cabal-3 tree and the compiler tree would be updated in tandem, and then the "freeze" would sort of apply to both in tandem as well. So there wouldn't be big changes after the freeze, but nor would the compiler be coupled to a _released_ lib. Rather, they would develop together, freeze together, and release together.

...

I don't disagree. But then we'd need to abandon any notion that versions of packages on Hackage and versions of packages in the GHC release tarball always match up. Might even be worth calling that out explicitly in the policy.

Not exactly. The tarball of the package on hackage should match the release tarball. Revisions don't change the tarball. They just add additional metadata to the index as well that cabal-install knows how to use in conjunction with the tarball: https://github.com/haskell-infra/hackage-trustees/blob/master/revisions-info...

--Gershom

Thanks for the feedback, Michael. Manuel, I believe you are also a Cabal-the-library consumer in Haskell For Mac? Michael, you brought up another problem tangentially related to the original integer-gmp issue but that was not in my original list earlier in this thread: * Cabal-2.0.0 had breaking changes in the API. This means that by association GHC itself broke BC, because it shipped with Cabal-2.0, without the usual grace period. Now, there are far fewer users of Cabal than of base. All, Michael in his previous email seems to be okay with breaking changes in Cabal given the conditions he stated (2 months grace period, advance notice of when the 2 months start). And perhaps this points to the lack of a need for the regular grace period applying to Cabal. How many other users of Cabal-the-library are there? In principle, every single Hackage package out there, which all have a Setup.hs script. Most of them are trivial, but how many did break because of these API changes? I for one am pretty happy for Cabal to move fast, but I'm concerned that these breaking changes happened without any kind of advance notice. To Simon's original point - there does not to be a clear policy and a good process surrounding Cabal itself and other GHC dependencies. So far we discussed mostly metadata changes, not API changes. And to be clear, folks did get some (post facto) notice in September: http://coldwa.st/e/blog/2017-09-09-Cabal-2-0.html. That's helpful, but I submit that in the future this really should be part of the GHC release announcement (which happened over a month before that), and in fact a migration guide circulated before the feature freeze, so downstream tooling authors can adapt. If this is not possible, then perhaps it's premature for GHC to include that given Cabal release. Again, GHC should always have the option to stick to the old Cabal version until things get ironed out. On 15 December 2017 at 08:42, Michael Snoyman wrote:

On Thu, Dec 14, 2017 at 12:27 PM, Boespflug, Mathieu wrote:

[snip]

...

* Or a middle ground: make feature freeze a thing. Meaning that for a couple of months before a major GHC release, the major new Cabal isn't technically released yet, but like GHC itself within this period, it's pretty staid, so not so much a moving target, and something downstream tooling authors can possibly adapt to even without any grace period on new metadata features. This assumes that the 2 months of feature freeze are enough time for downstream tooling. Thoughts from any of those maintainers?

Short answer: if there's a clear idea in advance of when this feature freeze is going to happen, I think we can coordinate releases of downstream tooling (Stack being the most important, but stackage-curator playing in as well) so that 2 months is sufficient. I'll talk with the rest of the Stack team to see if there are any concerns.

Longer answer: Stack intentionally avoids depending on the internals of Cabal wherever possible. Instead of calling library functions directly from within Haskell code to perform builds, for example, it interacts with the Setup.hs files over their command line interface.[1] This has two results:

* Stack can usually start using new GHC/Cabal versions without a new Stack release, since it's just shelling out for the actual build * There's not usually very much code churn needed in Stack to upgrade to a newer Cabal release

This past release was an exception because of all of the changes that landed, both the new cabal grammar to support the ^>= operator (making the old parser incapable of lossily parsing new files) and API changes (I think mostly around Backpack, though there was some code cleanup as well). In particular, the main interface we need from Cabal—the package description data types and parser—changed significantly enough that it took significant effort to upgrade. There were also new features added (like sub libraries and foreign libraries) that weren't immediately supported by the old Stack version, and had to be manually added in.

Tying this up: generally upgrading to a new Cabal release should be fine, and the only concern I'd have is fitting it into a release schedule with Stack. The complications that could slow that down are:

* Changes to the command line interface that Stack uses (hopefully those are exceedingly rare) * Major overhauls to the Stack-facing API

Michael

[1] This allows for more reproducible builds of older snapshots, insuring that the exact same Cabal library is performing the builds

Hi Mathieu, On 15 December 2017 at 08:41, Boespflug, Mathieu wrote:

In principle, every single Hackage package out there, which all have a Setup.hs script.

Also, the build-type: Simple packages (which are the vast majority on Hackage) are not affected at all, because they all use a default built-in setup script.

On 15 December 2017 at 16:32, Ben Gamari wrote:

"Boespflug, Mathieu" writes:

...

Thanks for the feedback, Michael.

Manuel, I believe you are also a Cabal-the-library consumer in Haskell For Mac?

Michael, you brought up another problem tangentially related to the original integer-gmp issue but that was not in my original list earlier in this thread:

* Cabal-2.0.0 had breaking changes in the API.

This means that by association GHC itself broke BC, because it shipped with Cabal-2.0, without the usual grace period.

I'm a bit confused; by "the usual grace period" do you mean the Core Library Committee's three release policy?

I did mean that one, yes. That was my question earlier - is Cabal along with *all* core libraries covered by the CLC's 3-release policy? The *Core Libraries* Committee (CLC) defines a "core library" as "Our definition of "core library" is a library that ships with GHC." (See https://wiki.haskell.org/Library_submissions#The_Libraries) But indeed, Cabal is not part of the CLC libraries list on that page. So I'm confused too: a) is Cabal a "core library", b) does that mean Cabal is bound by the 3-release policy?

GHC has never, as far as I am aware, considered major version bumps of its dependencies to be part of its interface. We perform a major bump of most libraries with nearly every release [1].

Yes, and major version bumps are not necessarily BC.

On 15 December 2017 at 19:03, Gershom B wrote:

On Fri, Dec 15, 2017 at 12:37 PM, Boespflug, Mathieu wrote:

...

I did mean that one, yes. That was my question earlier - is Cabal along with *all* core libraries covered by the CLC's 3-release policy?

The 3 release policy does not apply to all libraries maintained by the CLC. It applies to "basic libraries": https://prime.haskell.org/wiki/Libraries/3-Release-Policy

That clarifies it, thanks!

"Boespflug, Mathieu" writes:

Hi Gerhom,

On 14 December 2017 at 00:19, Gershom B wrote:

...

Mathieu:

I think the points about better tooling for documenting the correct claims in the release process are well taken. Updating the release notes manually leaves way too much room for error.

Indeed, the release notes have historically been a massive headache. Happily, I wrote a bit of the necessary tooling to fix this a few weeks ago [1].

[1] e4dc2cd51902a8cd83476f861cf52996e5adf157

...

However, I think you are incorrect that GHC 8.2.1 and 8.2.2 did not have cabal-install 1.24 support. They did. it works with them.

They did, and indeed Stack too worked just fine with them, but that was assuming that integer-gmp-1.0.1.0 really was what was shipped in the tarballs, not what it was on Hackage (until it got recently revised). I don't know which version of integer-gmp-1.0.1.0 was the intended one. They both have the same version number and neither seems more authoritative than the other to me. Had the Hackage one been the one that shipped, then I'm not sure that cabal-install-1.24 would have worked. Stack broke the moment what was on Hackage and what was in GHC bindists did not line up anymore. And with release notes mentioning incorrect version numbers, harder still to tell.

I agree that the cabal files uploaded to Hackage should match what is released or, if not, there should be a very good reason for divergence. [snip]

...

The general motivation of making a "feature freeze" more of a "freeze all the moving parts, really" I do agree with. Having a real freeze is part of a better release process, and it should allow all the downstream consumers of everything more time to really catch up. This is just one instance of that need.

Agreed.

Also agreed. Cheers, - Ben