I believe the standard policy would be to say that even master may only dependent on released versions of dependencies. That is after all the only way to have a master that is always ready to be cut for a release (as per modern CI practices). Given the tight coupling of some of the dependencies of GHC with GHC, maybe we need to consider something weaker, but I think, the weakest reasonable (from a CI standpoint) policy is to say that, while master may depend on pre-release versions, release branches can *only* depend on released dependencies. In other words, a release branch can only be cut when master has progressed to a point where it only depends on released versions of its dependencies. Under that compromise, cutting a GHC release branch may be delayed by a delayed upstream release, but hopefully the approx 3 month release process has enough slack to tolerate that. (It is obviously not ideal, though.) Cheers, Manuel PS: Simon, I am sorry, but IMHO it is too early for a summary and policy proposal as the discussion hasn’t really converged yet. In any case, I am happy to write a summary Trac page once we are there. Is that ok?

19.12.2017 06:41 Gershom B :

Let me try to formulate a synthetic policy as per Simon's request:

Policy: Bundled library maintainers agree to the following: 1) When GHC cuts a feature-freeze branch, they too (if anything has changed) cut a feature-freeze branch within two weeks at the maximum (ideally sooner), to be included in the main GHC freeze branch. If they do not do so, the last released version will be included instead. 2) When GHC releases the first release candidate, maintainers (if anything has changed) release new versions of their packages, to then be depended on directly in the GHC repo. All submodules are then replaced with their proper released versions for GHC release.

This policy can be enforced by GHC hq as part of the release process with the exception of a case in which there's coupling so that a new GHC _requires_ a new submodule release, and also the maintainer is not responsive. We'll have to deal with that case directly, likely by just appealing to the libraries committee or something to force a new release :-)

Motivation: This should help address multiple issues: 1) holdup of ghc on other releases. 2) lack of synchronization with ghc and other releases. 3) low lead-time for people to adapt to API changes in forthcoming library releases tied to ghc releases. In particular, because Cabal is part of this policy, it should help circumvent the sorts of problems that led to this thread initially. Further, because this only applies to freeze/release branches, it should not slow down rapid-implementation of cross-cutting changes more generally.

Cheers, Gershom _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Thanks for spelling this out Gershom. Reading it through, here are my questions: 1. What's the definition of "feature freeze"? Does it mean API stability? Does it mean not code changes at all except to fix a bug? Are performance fixes allowed in that case? 2. What's the minimum time between GHC cutting a feature-freeze branch and the first release candidate? And the minimum time between first release candidate and official release? Obviously, if each of these is 1 week (which I can't imagine would be the case), then these libraries could cut a feature-freeze branch after the official release, which obviously isn't intended. I apologize if these timings are already well established, I'm not familiar enough with GHC release cadence to know. I can't speak to GHC development itself, but from a downstream perspective, this sounds like the right direction. On Mon, Dec 18, 2017 at 9:41 PM, Gershom B wrote:

Let me try to formulate a synthetic policy as per Simon's request:

Policy: Bundled library maintainers agree to the following: 1) When GHC cuts a feature-freeze branch, they too (if anything has changed) cut a feature-freeze branch within two weeks at the maximum (ideally sooner), to be included in the main GHC freeze branch. If they do not do so, the last released version will be included instead. 2) When GHC releases the first release candidate, maintainers (if anything has changed) release new versions of their packages, to then be depended on directly in the GHC repo. All submodules are then replaced with their proper released versions for GHC release.

This policy can be enforced by GHC hq as part of the release process with the exception of a case in which there's coupling so that a new GHC _requires_ a new submodule release, and also the maintainer is not responsive. We'll have to deal with that case directly, likely by just appealing to the libraries committee or something to force a new release :-)

Motivation: This should help address multiple issues: 1) holdup of ghc on other releases. 2) lack of synchronization with ghc and other releases. 3) low lead-time for people to adapt to API changes in forthcoming library releases tied to ghc releases. In particular, because Cabal is part of this policy, it should help circumvent the sorts of problems that led to this thread initially. Further, because this only applies to freeze/release branches, it should not slow down rapid-implementation of cross-cutting changes more generally.

Cheers, Gershom _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Thanks for the terminology clarification, Gershom. As a general note, adding features or completing hitherto incomplete features on the release branch is something that should be avoided as much as possible. For starters, it is generally more work than doing it before the release branch is cut as things need to be added to master and the release branch, which slowly diverge. Moreover, the main purpose of the release branch is to stabilise the code base and to do so in a timely manner. As an example, if there is a new type system extension, but the error message are still bad, it may be allowed on the release branch with the understanding that the error messages are going to be fixed during the first month or so of the release branch lifecycle (i.e., the bulk of the work is done and tested before the release branch is cut). This in particular implies that everything merged to the release branch has to have been well tested (the wiki page states, ”[o]nly previously agreed on, stable and tested new functionality is allowed in.”) Implicitly adding new functionality by depending on a moving library dependency, which is out of the quality control of the GHC team, is not something that I would call ”stable and tested functionality". If we depend on the release version of a library on the release branch, we are still free to bump the library version if that is necessary to fix bugs (which may creep up due to GHC testing as you suggest) — I would expect that to be patch level bumps, though. If new library features are not sufficiently stable to move to that less tightly coupled process from T-3, then that feature set simply has to wait for the next GHC release (which by the new schedule will come much more quickly than before). Let me reiterate that the tighter release process is aimed at being able to achieve more frequent and more predictable GHC releases. This again ought to help the library development, too. In other words, we slow down at the right place to move faster overall (an old Kung Fu trick ;) Cheers, Manuel

20.12.2017, 04:28 Gershom B :

Good questions. In my below proposal I think I made an error in naming things. I checked back at the wiki page for the new release calendar schedule: https://ghc.haskell.org/trac/ghc/wiki/WorkingConventions/Releases/NewSchedul...

Based on that, what I was calling "freeze" is really just cutting the branch. But it isn't intended as a full freeze. That happens 3 months before release. The "feature freeze" in that calendar only comes with the first RC, 1 month before release.

I think that this timing still works with the proposal I have however -- bundled libs branch when GHC branches (T-3), and cut releases when GHC cuts the first RC (T-1). For bundled libs, I think we'd want to treat that branch (T-3) as closer to a feature freeze.

However, and here I disagree with Manuel, I think there's plenty of reason to _not_ cut release versions of libs at the time of the T-3 branch. In particular, due to the coupling, this may cause trouble if there are cross-cutting changes that need to be implemented for the sake of GHC working properly over the three month duration of the alpha. If there's a feature in a library designed to work in conjunction with a change in GHC, and that change in GHC needs to be altered in the course of the alpha (which may not be uncommon -- bug testing can often reveal such things) then it is likely the library may need to be changed too. I don't see any concrete goal solved in making this process significantly more difficult. I thought Moritz' examples in this thread were very revealing with regards to such possibilities. It is not clear what cost function the stronger proposal is trying to optimize for.

If it is that we want a branch that is "always ready to be cut for release" (why? is such a thing even possible anytime in the foreseeable future?), one middle ground may be to cut _candidate_ releases of bundled libs on branch (T-3).

--Gershom

On Tue, Dec 19, 2017 at 10:12 AM, Michael Snoyman wrote:

...

Thanks for spelling this out Gershom. Reading it through, here are my questions:

1. What's the definition of "feature freeze"? Does it mean API stability? Does it mean not code changes at all except to fix a bug? Are performance fixes allowed in that case? 2. What's the minimum time between GHC cutting a feature-freeze branch and the first release candidate? And the minimum time between first release candidate and official release? Obviously, if each of these is 1 week (which I can't imagine would be the case), then these libraries could cut a feature-freeze branch after the official release, which obviously isn't intended. I apologize if these timings are already well established, I'm not familiar enough with GHC release cadence to know.

I can't speak to GHC development itself, but from a downstream perspective, this sounds like the right direction.

On Mon, Dec 18, 2017 at 9:41 PM, Gershom B wrote:

...

Let me try to formulate a synthetic policy as per Simon's request:

Policy: Bundled library maintainers agree to the following: 1) When GHC cuts a feature-freeze branch, they too (if anything has changed) cut a feature-freeze branch within two weeks at the maximum (ideally sooner), to be included in the main GHC freeze branch. If they do not do so, the last released version will be included instead. 2) When GHC releases the first release candidate, maintainers (if anything has changed) release new versions of their packages, to then be depended on directly in the GHC repo. All submodules are then replaced with their proper released versions for GHC release.

This policy can be enforced by GHC hq as part of the release process with the exception of a case in which there's coupling so that a new GHC _requires_ a new submodule release, and also the maintainer is not responsive. We'll have to deal with that case directly, likely by just appealing to the libraries committee or something to force a new release :-)

Motivation: This should help address multiple issues: 1) holdup of ghc on other releases. 2) lack of synchronization with ghc and other releases. 3) low lead-time for people to adapt to API changes in forthcoming library releases tied to ghc releases. In particular, because Cabal is part of this policy, it should help circumvent the sorts of problems that led to this thread initially. Further, because this only applies to freeze/release branches, it should not slow down rapid-implementation of cross-cutting changes more generally.

Cheers, Gershom _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

This proposal seems to have trailed off. I think it is worth putting a bow on it and getting it signed off on for the GHC release policy page. Let me restate it: Bundled library maintainers agree to the following responsibilities: 1) When GHC cuts a feature-freeze branch, they too (if anything has changed) cut a feature-freeze branch within two weeks at the maximum (ideally sooner), to be included in the main GHC freeze branch. If they do not do so, the last released version will be included instead. 2) When GHC releases the first release candidate, maintainers (if anything has changed) release new versions of their packages, to then be depended on directly in the GHC repo. All submodules are then replaced with their proper released versions for GHC release. I know Chak and I had a back and forth, but I consider it resolved in the direction of this proposal at this point, as a strict improvement over the current situation that does not seem to invite further controversy. If in the event of future things we learn, this proposal is inadequate, we can revisit then. The one thing missing from both this proposal and the current release policy page in general is discussion of the head overlay to hackage. I think discussion of this should be integrated in both as a separate issue. In particular — the feature freeze branches of bundled libraries can and should be regularly updated in the head overlay. Thoughts? —Gershom On December 19, 2017 at 8:53:22 PM, Gershom B (gershomb@gmail.com) wrote: On December 19, 2017 at 8:15:49 PM, Manuel M T Chakravarty ( chak@justtesting.org) wrote:

...

This in particular implies that everything merged to the release branch has to have been well tested (the wiki page states, ”[o]nly previously agreed on, stable and tested new functionality is allowed in.”)

Ok — here’s where I get confused. In my experience, “well tested” is a relative term. Certainly if we want a three month period between branch and release, that’s because we’re expecting a _fair amount_ of bugs to shake out over that period. As you point out, the wiki page says “agreed on, stable and tested.” But nonetheless — that’s leaving a fair amount of room for bugs to be found. Since there’s a period when features are written, and another in which focus is all on bugfixing, then this is to be expected. Keeping moving the barrier for making it into even a _branch_ higher and higher seems weird to me. We’re already creating a new set of significant barriers by even starting to have this branch process at all.

Implicitly adding new functionality by depending on a moving library dependency, which is out of the quality control of the GHC team, is not something that I would call ”stable and tested functionality”.

And now here’s where you lose me entirely. The point is you wouldn’t be depending on a moving library dependency. You’d be depending on a _release branch_ just the same as GHC. Unless you mean to say that we shouldn’t trust the maintainers of bundled packages to only put “stable and tested” features in to a release branch? In the abstract, sure. If we were just including random github repos sourced from stackoverflow answers, sure. But in the actual concrete history of GHC and Haskell development — we know the maintainers of these libs. We know they move slowly in general, and they have processes too. And we know that if these libs, which are hard to upgrade on their own (because they’re bundled with GHC), are going to get a wide exposure to not just internal tests, but external users, it will be via being bundled with GHC. Since I would expect, based on past experience, that bundled libraries will often need small bugfixes and tweaks, as you agree, then what you’ve just sort of proposed is that we release extra versions of them that end up not bundled with GHC versions, and which are less well tested. What is the difference between cutting a branch and depending on it, which is what I propose, and what you propose, except that we’ll end up with extra version bumps in your scheme, and hackage packages with bugs that haven’t had a chance to be shaken out, and may, in some cases (integer-gmp, ghc-prim, base), not even make _sense_ outside of a GHC release. I understand that its not pleasant, but GHC and the bundled packages have, for the time being, a very special relationship. We can’t decouple them by administrative fiat. GHC depends on them, and they also need GHC to run. So their lifecycles are intertwined, and their development processes are intertwined. This is how they have been developed and maintained for years. Why do we suddenly want to pretend that now these maintainers are “out of the quality control of the GHC team” like they’re foreign entities? I also want to underline this is not about “new library features” as such — this is about cross-cutting development of the sort Moritz described that requires modifying GHC and bundled libs both. Another element of the two-way character of this relationship is that when GHC develops a new feature, often it is the bundled libs that are the first to take advantage of it — and this is a good thing, as it allows further stress-testing of the feature. (And sometimes, with certain deprication warnings, etc., the feature is even especially _for_ those libs). But if the feature turns out to need tweaks in the stability period (and this is far from unheard of) then it is the bundled libs that need to change to adapt. If you have already pinned yourself to released versions of libs, this is very hard! It would be nice if bundled libs were upstream of GHC, but they’re not, so we should stop pretending. Everyone’s in the same stream together, and everyone will have to agree to row :-) Again — what is objectionable to having a release branch cut instead of a full release at T-3? What danger from this do you anticipate? Personally, I see only upside. More flexibility — same mitigation of risk, same alignment of processes. We already have a number of changes in the pipeline, not least the new schedule in any form. How about we try that, along with a more modest proposal for bundled libs, and just _see how it feels_ before making any futher decisions. A “big bang” in process doesn’t need to be bigger — that only introduces more risk factors that will prevent us from evaluating the first changeset independently. Honestly, before making any further policy I’d just like to see if one or two releases can be done with the new schedule at all, at which point people will better be able to judge what the remaining pain points really are, instead of speculating about other future problems. —Gershom