Thanks for this debate. (And thank you Austin for provoking it by articulating a medium term plan.) Our intent has always been that that the latest version on each branch is solid. There have been one or two occasions when we have knowingly abandoned a dodgy release branch entirely, but not many. So I think the major trick we are missing is this: We don't know what the show-stopping bugs on a branch are For example, here are three responses to Austin's message: | The only potential issue here is that not a single 7.8 release will be | able to bootstrap LLVM-only targets due to #9439. I'm not sure how | 8960 looks rather serious and potentially makes all of 7.8 a no-go | for some users. | We continue to use 7.2, at least partly because all newer versions of | ghc have had significant bugs that affect us That's not good. Austin's message said about 7.8.4 "No particular pressure on any outstanding bugs to release immediately". There are several dozen tickets queued up on 7.8.4 (see here https://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.8.4), but 95% of them are "nice to have". So clearly the message is not getting through. My conclusion * I think we (collectively!) should make a serious attempt to fix show-stopping bugs on a major release branch. (I agree that upgrading to the next major release often simply brings in a new wave of bugs because of GHC's rapid development culture.) * We can only possibly do this if a) we can distinguish "show-stopping" from "nice to have" b) we get some help (thank you John Lato for implicitly offering) I would define a "show-stopping" bug as one that simply prevents you from using the release altogether, or imposes a very large cost at the user end. For mechanism I suggest this. On the 7.8.4 status page (or in general, on the release branch page you want to influence), create a section "Show stoppers" with a list of the show-stopping bugs, including some English-language text saying who cares so much and why. (Yes I know that it might be there in the ticket, but the impact is much greater if there is an explicit list of two or three personal statements up front.) Concerning 7.8.4 itself, I think we could review the decision to abandon it, in the light of new information. We might, for example, fix show-stoppers, include fixes that are easy to apply, and not-include other fixes that are harder. Opinions? I'm not making a ruling here! Simon | -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of Ben | Gamari | Sent: 04 October 2014 04:52 | To: Austin Seipp; ghc-devs@haskell.org | Cc: Simon Marlow | Subject: Re: Tentative high-level plans for 7.10.1 | | Austin Seipp writes: | | snip. | | > | > We do not believe we will ship a 7.8.4 at all, contrary to what you | > may have seen on Trac - we never decided definitively, but there is | > likely not enough time. Over the next few days, I will remove the | > defunct 7.8.4 milestone, and re-triage the assigned tickets. | > | The only potential issue here is that not a single 7.8 release will be | able to bootstrap LLVM-only targets due to #9439. I'm not sure how | much of an issue this will be in practice but there should probably be | some discussion with packagers to ensure that 7.8 is skipped on | affected platforms lest users be stuck with no functional stage 0 | compiler. | | Cheers, | | - Ben

I re-targeted some of the bugs that were "obviously" the same SpecConstr issue to 7.8.4. There are a few others that should probably also be re-targeted, but I couldn't tell from a quick scan of the long comment threads. Looking at the 7.8.4 status page, it's now quite clear that the SpecConstr bug is a show stopper i.e. it affects lots of people/core libraries and doesn't really have a good workaround, as turning of SpecConstr will most likely make e.g. vector too slow. On Tue, Oct 7, 2014 at 11:23 AM, Johan Tibell wrote:

On Tue, Oct 7, 2014 at 10:12 AM, Simon Peyton Jones

...

wrote:

...

| 8960 looks rather serious and potentially makes all of 7.8 a no-go | for some users.

I think this is the big issue. If you look at all the related bugs linked from #8960, lots of users are affected. I think this bug alone probably warrants a release. We should also move all those related bugs to the 7.8.4 milestone, so the impact of this issue is more clear.

...

My conclusion

* I think we (collectively!) should make a serious attempt to fix show-stopping bugs on a major release branch. (I agree that upgrading to the next major release often simply brings in a new wave of bugs because of GHC's rapid development culture.)

* We can only possibly do this if a) we can distinguish "show-stopping" from "nice to have" b) we get some help (thank you John Lato for implicitly offering)

All sounds good to me. I can help with backporting bug fixes if needed. In return I would encourage people to not mix bug fixes with "I rewrote the compiler" commits. :)

I would define a "show-stopping" bug as one that simply prevents you from

...

using the release altogether, or imposes a very large cost at the user end.

Agreed.

-- Johan

First off, I just wanted to tell everyone - thank you for the feedback! I actually left these tickets in their place/milestones just in case something like this popped up, so I wouldn't have to undo it later. It seems like there's actually a fair amount of support for 7.8.4, where before we didn't get much of an indication as to user needs. As a result, I'll be leaving the 7.8.4 milestone tickets, but will still cull them down to what's acceptable, and we'll aim for those. #8960 seems to be the main one. As I said in the initial email, I'll follow up on this shortly after this, later today. On Tue, Oct 7, 2014 at 6:46 AM, Mikolaj Konarski wrote:

...

Our intent has always been that that the latest version on each branch is solid. There have been one or two occasions when we have knowingly abandoned a dodgy release branch entirely, but not many.

Perhaps we could do the opposite. Announce beforehand that a release branch X is going to be LTS (of Very Stable Release; roughly 1 in 4 branches?) and so very few major new features will be included in the release X+1 (there is just not enough time for both, as Austin explained). Then, on the GHC maintainers' side, put off accepting any "I rewrote the compiler" commits into HEAD for long time. On the community side, focus on bug fixes and non-disruptive, incremental improvements. Avoid API changes, whatever that means. Update release X many times, until very stable.

A more radical proposal would be to do the above, but announce that X+! is going to be Very Stable Release and accept no major new features into HEAD at all and even revert any minor new features just before X+1 release, if non-trivial bugs in them are discovered. Then release X can be abandoned quickly, knowing that X+! will most probably resolve any problems in X, without introducing new ones.

In either case, the main point is the announcement and so the focus of the community on bug-fixing and keeping HEAD close to the named releases, to make bug-fixing and back- and forward- porting easy. _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

-- Regards, Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

Ok, if the ghc devs decide to do a 7.8.4 release, I will explicitly commit to helping backport patches. However, I don't know how to do so. Therefore, I'm going to ask Austin (as he's probably the most knowledgeable) to update the 7.8.4 wiki page with the process people should use to contribute backports. I'm guessing it's probably something like this: checkout the 7.8.4 release branch (which branch is it? ghc-7.8?) git cherry-pick the desired commit(s) ? (make a phab request for ghc-hq to review?) update Trac with what you've done (or if this is already documented somewhere, please point me to it). Unfortunately this doesn't have any way of showing that I'm working on a specific backport/merge, so there's potential for duplicate work, which isn't great. I also agree with Nicolas that it's likely possible to make better use of git to help with this sort of work, but that's a decision for ghc hq so I won't say any more on that. Cheers, John On Tue, Oct 7, 2014 at 4:12 PM, Simon Peyton Jones wrote:

Thanks for this debate. (And thank you Austin for provoking it by articulating a medium term plan.)

Our intent has always been that that the latest version on each branch is solid. There have been one or two occasions when we have knowingly abandoned a dodgy release branch entirely, but not many.

So I think the major trick we are missing is this:

We don't know what the show-stopping bugs on a branch are

For example, here are three responses to Austin's message:

| The only potential issue here is that not a single 7.8 release will be | able to bootstrap LLVM-only targets due to #9439. I'm not sure how

| 8960 looks rather serious and potentially makes all of 7.8 a no-go | for some users.

| We continue to use 7.2, at least partly because all newer versions of | ghc have had significant bugs that affect us

That's not good. Austin's message said about 7.8.4 "No particular pressure on any outstanding bugs to release immediately". There are several dozen tickets queued up on 7.8.4 (see here https://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.8.4), but 95% of them are "nice to have".

So clearly the message is not getting through.

My conclusion

* I think we (collectively!) should make a serious attempt to fix show-stopping bugs on a major release branch. (I agree that upgrading to the next major release often simply brings in a new wave of bugs because of GHC's rapid development culture.)

* We can only possibly do this if a) we can distinguish "show-stopping" from "nice to have" b) we get some help (thank you John Lato for implicitly offering)

I would define a "show-stopping" bug as one that simply prevents you from using the release altogether, or imposes a very large cost at the user end.

For mechanism I suggest this. On the 7.8.4 status page (or in general, on the release branch page you want to influence), create a section "Show stoppers" with a list of the show-stopping bugs, including some English-language text saying who cares so much and why. (Yes I know that it might be there in the ticket, but the impact is much greater if there is an explicit list of two or three personal statements up front.)

Concerning 7.8.4 itself, I think we could review the decision to abandon it, in the light of new information. We might, for example, fix show-stoppers, include fixes that are easy to apply, and not-include other fixes that are harder.

Opinions? I'm not making a ruling here!

Simon

| -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of Ben | Gamari | Sent: 04 October 2014 04:52 | To: Austin Seipp; ghc-devs@haskell.org | Cc: Simon Marlow | Subject: Re: Tentative high-level plans for 7.10.1 | | Austin Seipp writes: | | snip. | | > | > We do not believe we will ship a 7.8.4 at all, contrary to what you | > may have seen on Trac - we never decided definitively, but there is | > likely not enough time. Over the next few days, I will remove the | > defunct 7.8.4 milestone, and re-triage the assigned tickets. | > | The only potential issue here is that not a single 7.8 release will be | able to bootstrap LLVM-only targets due to #9439. I'm not sure how | much of an issue this will be in practice but there should probably be | some discussion with packagers to ensure that 7.8 is skipped on | affected platforms lest users be stuck with no functional stage 0 | compiler. | | Cheers, | | - Ben

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

On 2014-10-08 at 02:13:01 +0200, Carter Schonwald wrote:

the checkout process for the 7.8 branch is a bit involved (and NB: you really want to use a different tree than one for working on head, the checkout process is different )

$ git clone -b ghc-7.8 git://git.haskell.org/ghc.git ghc-7.8TREE $ cd ghc-7.8TREE/ $ ./sync-all get -b ghc-7.8

(theres no need for a lot of this with HEAD)

Just to clarify/remind why this is needed: The GHC 7.8 branch was not converted to a proper submodule-only scheme like GHC HEAD was. Unless we keep maintaining GHC 7.8 for longer than a 7.8.4 release, this irregularity will become less of a concern, as the stable GHC 7.10 branch will be switchable to/from later branches such as GHC 7.12/HEAD w/o requiring a separately cloned tree. However, should GHC 7.8.x turn out to become a LTS-ishly maintained branch, we may want to consider converting it to a similiar Git structure as GHC HEAD currently is, to avoid having to keep two different sets of instructions on the GHC Wiki for how to work on GHC 7.8 vs working on GHC HEAD/7.10 and later. Cheers, hvr

Speaking for myself, I don't think the question of doing a 7.8.4 release at all needs to be entangled with the LTS issue. On Wed, Oct 8, 2014 at 8:23 AM, Edward Z. Yang wrote:

Excerpts from Herbert Valerio Riedel's message of 2014-10-08 00:59:40 -0600:

...

However, should GHC 7.8.x turn out to become a LTS-ishly maintained branch, we may want to consider converting it to a similiar Git structure as GHC HEAD currently is, to avoid having to keep two different sets of instructions on the GHC Wiki for how to work on GHC 7.8 vs working on GHC HEAD/7.10 and later.

Emphatically yes. Lack of submodules on the 7.8 branch makes working with it /very/ unpleasant.

Edward _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

I agree a section show stoppers is a good idea, in parallel would it make sense to use the priority "highest" for tickets that we consider showstoppers? Austin did a great of explaining the difficulties of backporting fixes, my reaction is that we have to have higher quality releases so that ideally we have 0 backports. Having a showstoppers section will help that but I think we need to work harder at getting volunteers to write tests. For most people that's not exciting but it is a good way to get started on helping and would be an immense help in producing higher quality releases. As Austin also pointed out things change rapidly, it's hard to keep up and it's getting harder for people to get to the point where they feel they are decent Haskell programmers. So in addition to testing it would be great if we could get more people to document, i.e. write tutorials etc. It is difficult to balance being a research language and being a viable language for industrial use. FWIW, I personally feel that we side too much on being a research language. On Tue, Oct 7, 2014 at 5:12 AM, Simon Peyton Jones wrote:

Thanks for this debate. (And thank you Austin for provoking it by articulating a medium term plan.)

Our intent has always been that that the latest version on each branch is solid. There have been one or two occasions when we have knowingly abandoned a dodgy release branch entirely, but not many.

So I think the major trick we are missing is this:

We don't know what the show-stopping bugs on a branch are

For example, here are three responses to Austin's message:

| The only potential issue here is that not a single 7.8 release will be | able to bootstrap LLVM-only targets due to #9439. I'm not sure how

| 8960 looks rather serious and potentially makes all of 7.8 a no-go | for some users.

| We continue to use 7.2, at least partly because all newer versions of | ghc have had significant bugs that affect us

That's not good. Austin's message said about 7.8.4 "No particular pressure on any outstanding bugs to release immediately". There are several dozen tickets queued up on 7.8.4 (see here https://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.8.4), but 95% of them are "nice to have".

So clearly the message is not getting through.

My conclusion

* I think we (collectively!) should make a serious attempt to fix show-stopping bugs on a major release branch. (I agree that upgrading to the next major release often simply brings in a new wave of bugs because of GHC's rapid development culture.)

* We can only possibly do this if a) we can distinguish "show-stopping" from "nice to have" b) we get some help (thank you John Lato for implicitly offering)

I would define a "show-stopping" bug as one that simply prevents you from using the release altogether, or imposes a very large cost at the user end.

For mechanism I suggest this. On the 7.8.4 status page (or in general, on the release branch page you want to influence), create a section "Show stoppers" with a list of the show-stopping bugs, including some English-language text saying who cares so much and why. (Yes I know that it might be there in the ticket, but the impact is much greater if there is an explicit list of two or three personal statements up front.)

Concerning 7.8.4 itself, I think we could review the decision to abandon it, in the light of new information. We might, for example, fix show-stoppers, include fixes that are easy to apply, and not-include other fixes that are harder.

Opinions? I'm not making a ruling here!

Simon

| -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of Ben | Gamari | Sent: 04 October 2014 04:52 | To: Austin Seipp; ghc-devs@haskell.org | Cc: Simon Marlow | Subject: Re: Tentative high-level plans for 7.10.1 | | Austin Seipp writes: | | snip. | | > | > We do not believe we will ship a 7.8.4 at all, contrary to what you | > may have seen on Trac - we never decided definitively, but there is | > likely not enough time. Over the next few days, I will remove the | > defunct 7.8.4 milestone, and re-triage the assigned tickets. | > | The only potential issue here is that not a single 7.8 release will be | able to bootstrap LLVM-only targets due to #9439. I'm not sure how | much of an issue this will be in practice but there should probably be | some discussion with packagers to ensure that 7.8 is skipped on | affected platforms lest users be stuck with no functional stage 0 | compiler. | | Cheers, | | - Ben

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

I think we need to work harder at getting volunteers to write tests it would be great if we could get more people to document, i.e. write tutorials Good ideas, thank you. It would be great if you felt able to contribute to one or the other (or both) yourself. Simon From: George Colpitts [mailto:george.colpitts@gmail.com] Sent: 08 October 2014 01:35 To: Simon Peyton Jones Cc: Ben Gamari; Austin Seipp; ghc-devs@haskell.org; Simon Marlow Subject: Re: Tentative high-level plans for 7.10.1 I agree a section show stoppers is a good idea, in parallel would it make sense to use the priority "highest" for tickets that we consider showstoppers? Austin did a great of explaining the difficulties of backporting fixes, my reaction is that we have to have higher quality releases so that ideally we have 0 backports. Having a showstoppers section will help that but I think we need to work harder at getting volunteers to write tests. For most people that's not exciting but it is a good way to get started on helping and would be an immense help in producing higher quality releases. As Austin also pointed out things change rapidly, it's hard to keep up and it's getting harder for people to get to the point where they feel they are decent Haskell programmers. So in addition to testing it would be great if we could get more people to document, i.e. write tutorials etc. It is difficult to balance being a research language and being a viable language for industrial use. FWIW, I personally feel that we side too much on being a research language. On Tue, Oct 7, 2014 at 5:12 AM, Simon Peyton Jones mailto:simonpj@microsoft.com> wrote: Thanks for this debate. (And thank you Austin for provoking it by articulating a medium term plan.) Our intent has always been that that the latest version on each branch is solid. There have been one or two occasions when we have knowingly abandoned a dodgy release branch entirely, but not many. So I think the major trick we are missing is this: We don't know what the show-stopping bugs on a branch are For example, here are three responses to Austin's message: | The only potential issue here is that not a single 7.8 release will be | able to bootstrap LLVM-only targets due to #9439. I'm not sure how | 8960 looks rather serious and potentially makes all of 7.8 a no-go | for some users. | We continue to use 7.2, at least partly because all newer versions of | ghc have had significant bugs that affect us That's not good. Austin's message said about 7.8.4 "No particular pressure on any outstanding bugs to release immediately". There are several dozen tickets queued up on 7.8.4 (see here https://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.8.4), but 95% of them are "nice to have". So clearly the message is not getting through. My conclusion * I think we (collectively!) should make a serious attempt to fix show-stopping bugs on a major release branch. (I agree that upgrading to the next major release often simply brings in a new wave of bugs because of GHC's rapid development culture.) * We can only possibly do this if a) we can distinguish "show-stopping" from "nice to have" b) we get some help (thank you John Lato for implicitly offering) I would define a "show-stopping" bug as one that simply prevents you from using the release altogether, or imposes a very large cost at the user end. For mechanism I suggest this. On the 7.8.4 status page (or in general, on the release branch page you want to influence), create a section "Show stoppers" with a list of the show-stopping bugs, including some English-language text saying who cares so much and why. (Yes I know that it might be there in the ticket, but the impact is much greater if there is an explicit list of two or three personal statements up front.) Concerning 7.8.4 itself, I think we could review the decision to abandon it, in the light of new information. We might, for example, fix show-stoppers, include fixes that are easy to apply, and not-include other fixes that are harder. Opinions? I'm not making a ruling here! Simon | -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces@haskell.orgmailto:ghc-devs-bounces@haskell.org] On Behalf Of Ben | Gamari | Sent: 04 October 2014 04:52 | To: Austin Seipp; ghc-devs@haskell.orgmailto:ghc-devs@haskell.org | Cc: Simon Marlow | Subject: Re: Tentative high-level plans for 7.10.1 | | Austin Seipp mailto:austin@well-typed.com> writes: | | snip. | | > | > We do not believe we will ship a 7.8.4 at all, contrary to what you | > may have seen on Trac - we never decided definitively, but there is | > likely not enough time. Over the next few days, I will remove the | > defunct 7.8.4 milestone, and re-triage the assigned tickets. | > | The only potential issue here is that not a single 7.8 release will be | able to bootstrap LLVM-only targets due to #9439. I'm not sure how | much of an issue this will be in practice but there should probably be | some discussion with packagers to ensure that 7.8 is skipped on | affected platforms lest users be stuck with no functional stage 0 | compiler. | | Cheers, | | - Ben _______________________________________________ ghc-devs mailing list ghc-devs@haskell.orgmailto:ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Speaking as a user, I think Johan's concern is well-founded. For us, ghc-7.8.3 was the first of the 7.8 line that was really usable in production, due to #8960 and other bugs. Sure, that can be worked around in user code, but it takes some time for developers to locate the issues, track down the bug, and implement the workaround. And even 7.8.3 has some bugs that cause minor annoyances (either ugly workarounds or intermittent build failures that I haven't had the time to debug); it's definitely not solid. Similarly, 7.6.3 was the first 7.6 release that we were able to use in production. I'm particularly concerned about ghc-7.10 as the AMP means there will be significant lag in identifying new bugs (since it'll take time to update codebases for that major change). For the curious, within the past few days we've seen all the following, some multiple times, all so far intermittent:

ghc: panic! (the 'impossible' happened) (GHC version 7.8.3.0 for x86_64-unknown-linux): kindFunResult ghc-prim:GHC.Prim.*{(w) tc 34d}

ByteCodeLink.lookupCE During interactive linking, GHCi couldn't find the following symbol: some_mangled_name_closure

ghc: mmap 0 bytes at (nil): Invalid Argument

internal error: scavenge_one: strange object 2022017865

Some of these I've mapped to likely ghc issues, and some are fixed in HEAD, but so far I haven't had an opportunity to put together reproducible test cases. And that's just bugs that we haven't triaged yet, there are several more for which workarounds are in place. John L. On Sat, Oct 4, 2014 at 2:54 PM, Johan Tibell wrote:

On Fri, Oct 3, 2014 at 11:35 PM, Austin Seipp wrote:

...

- Cull and probably remove the 7.8.4 milestone. - Simply not enough time to address almost any of the tickets in any reasonable timeframe before 7.10.1, while also shipping them. - Only one, probably workarouadble, not game-changing bug (#9303) marked for 7.8.4. - No particular pressure on any outstanding bugs to release immediately. - ANY release would be extremely unlikely, but if so, only backed by the most critical of bugs. - We will move everything in 7.8.4 milestone to 7.10.1 milestone. - To accurately catalogue what was fixed. - To eliminate confusion.

#8960 looks rather serious and potentially makes all of 7.8 a no-go for some users. I'm worried that we're (in general) pushing too many bug fixes towards future major versions. Since major versions tend to add new bugs, we risk getting into a situation where no major release is really solid.

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

On 2014-10-06 at 11:03:19 +0200, p.k.f.holzenspies@utwente.nl wrote: [...]

The idea behind an LTS-GHC would be to continue bug-fixing on the LTS-version, even if newer major versions no longer get bug-fixing support. To some extent, there will be redundancies (bugs that have disappeared in newer versions because newer code does the same and more, still needing to be fixed on the LTS code base), but the upside would be a clear prioritisation between stability (LTS) and innovation (latest major release).

As I'm not totally sure what you mean: Assuming we already had decided years ago to follow LTS-style, given GHC 7.0, 7.2, 7.4, 7.6, 7.8 and the future 7.10; which of those GHC versions would you have been considered a LTS version? [...]

The danger, of course, is that people aren't very enthusiastic about bug-fixing older versions of a compiler, but for language/compiler-uptake, this might actually be a Better Way.

Maybe some of the commercial GHC users might be interested in donating the manpower to maintain older GHC versions. It's mostly a time-consuming QA & auditing process to maintain old GHCs.

On Mon, 2014-10-06 at 11:38 +0200, Johan Tibell wrote:

On Mon, Oct 6, 2014 at 11:28 AM, Herbert Valerio Riedel wrote:

...

On 2014-10-06 at 11:03:19 +0200, p.k.f.holzenspies@utwente.nl wrote:

...

The danger, of course, is that people aren't very enthusiastic about bug-fixing older versions of a compiler, but for language/compiler-uptake, this might actually be a Better Way.

Maybe some of the commercial GHC users might be interested in donating the manpower to maintain older GHC versions. It's mostly a time-consuming QA & auditing process to maintain old GHCs.

What can we do to make that process cheaper? In particular, which are the manual steps in making a new GHC release today?

In the long run back porting bugfixes is the route successful OSS projects take. Once people have written large enough Haskell programs they will stop jumping onto the newer version all the time and will demand backports of bug fixes. This is already happening to some extent in cabal (as cabal is tied to a ghc release which means we need to backport changes sometimes.)

Quick note/experience report: In my experience maintaining a project which has several older/supported 'production' versions, using backports is counter-productive, and we decided to manage things the other way around (which is fairly easy to do thanks to git and git-imerge): Instead of fixing things in the latest version of the product, we fix things in the *oldest* version of the product which should contain the fix (or feature, whatever). Then we regularly forward-merge version branches into the next version (especially when a release is made). So, if 1.4.x, 1.5.x, 1.6.x and 1.7.x are 'supported' versions, and some bug is found in 1.6.2, but turns out to be introduced in 1.5.1, we fix the bug in the 1.5 branch. Then, if the bugfix is important enough, we merge 1.4 in 1.5 (which can be a no-op), 1.5 in 1.6, and 1.6 into 1.7. As such, every version branch 'contains' all 'older' branches. Even though the codebase can diverge quite a bit between the 1.5 and 1.7 tree, these merges tend to be fairly easy, especially since the required changes are split between the 1.5-1.6 merge and the 1.6-1.7 merge. 'Small' commits with clear purpose, minimal changes & clear commit messages tend to help a lot in this process as well (especially when using git-imerge). The squashing performed by Phab as mentioned in some other messages on the list can be troublesome in this regard... (I'm not a big fan of squash merges myself, from a maintainer POV). Nicolas

Hello Daniel, On Mon, 2014-10-06 at 12:46 +0200, Daniel Trstenjak wrote:

...

So, if 1.4.x, 1.5.x, 1.6.x and 1.7.x are 'supported' versions, and some bug is found in 1.6.2, but turns out to be introduced in 1.5.1, we fix the bug in the 1.5 branch.

Then, if the bugfix is important enough, we merge 1.4 in 1.5 (which can be a no-op), 1.5 in 1.6, and 1.6 into 1.7. As such, every version branch 'contains' all 'older' branches.

I don't like this practise, because you certainly don't want to always incorprocate all commits of one release branch into an other.

Just think about a hackish bug fix needed to be added in a former release, and in a newer release the problem has been solved in a completely different way, and now, if you have bad luck, the former release branch merges without conflicts into the new one, now getting the hackish fix into the new release, which might be even harmful.

Agree, although I think this is less of an issue in practice because we enforce code reviews for all commits, including 'merge' commits, even if the merge was 100% automatic (hence we have PRs for '1.5-for-1.6' branches once in a while). These 'workarounds' are spotted easily during this process. Next to that, chances are fairly low a 'hack' won't in any way conflict with a 'proper fix', since they tend to touch (a portion of) the same code most of the time (except in build systems maybe). Using 'git-imerge' helps here quite a bit as well, since the conflicts aren't buried between 100s of unrelated changes (like what git-merge does).

IMHO using cherry picking in this case is a lot better manageable.

Yet it has a (IMHO) major drawback: it requires a system next to VCS (issue tracker or alike) to make sure all fixes are propagated to all applicable versions, which all too often results in (human) error. "Hey, we reported this against 1.6.3, and it was fixed in 1.6.4, but now we upgraded to 1.7.3 which was released after 1.6.4 and the bug is back" is no good PR. Anyway, it's not like I intend to push GHC development/maintenance in any specific direction at all, just wanted to provide some experience report :-) Regards, Nicolas

The steps for making a GHC release are here: https://ghc.haskell.org/trac/ghc/wiki/MakingReleases So, for the record, making a release is not *that* arduous, but it does take time. On average it will take me about 1 day or so to go from absolutely-nothing to release announcement: 1. Bump version, update configure.ac, tag. 2. Build source tarball (this requires 1 build, but can be done very quickly). 3. Make N binary builds for each platform (the most time consuming part, as this requires heavy optimizations in the builds). 4. Upload documentation for all libraries. 5. Update webpage and upload binaries. 6. Send announcement. 7. Upload binaries from other systems later. Herbert has graciously begun taking care of stewarding and uploading the libraries. So, there are a few steps we could introduce to alleviate this process technically in a few ways, but ultimately all of these have to happen, pretty much (regardless of the automation involved). But I don't think this is the real problem. The real problem is that GHC moves forward in terms of implementation extremely, extremely quickly. It is not clear how to reconcile this development pace with something like needing dozens of LTS releases for a stable version. At least, not without a lot of concentrated effort from almost every single developer. A lot of it can be alleviated through social process perhaps, but it's not strictly technical IMO. What do I mean by that? I mean that: - We may introduce a feature in GHC version X.Y - That might have a bug, or other problems. - We may fix it, and in the process, fix up a few other things and refactor HEAD, which will be GHC X.Y+2 eventually. - Repeat steps 2-3 a few times. - Now we want to backport the fixes for that feature in HEAD back to X.Y. - But GHC X.Y has *significantly* diverged from HEAD in that timeframe, because of step 3 being repeated! In other words: we are often so aggressive at refactoring code that the *act* of backporting in and of itself can be complicated, and it gets harder as time goes on - because often the GHC of a year ago is so much different than the GHC of today. As a concrete example of this, let's look at the changes between GHC 7.8.2 and GHC 7.8.3: https://github.com/ghc/ghc/compare/ghc-7.8.2-release...ghc-7.8.3-release There are about ~110 commits between 7.8.2 and 7.8.3. But as the 7.8 branch lived on, backporting fixes became significantly more complex. In fact, I estimate close to 30 of those commits were NOT direct 7.8 requirements - but they were brought in because _actual fixes_ were dependent on them, in non-trivial ways. Take for example f895f33 by Simon PJ, which fixes #9023. The problem with f895f33 is that by the time we fixed the bug in HEAD with that commit, the history had changed significantly from the branch. In order to get f895f33 to plant easily, I had to backport *at least* 12 to 15 other commits, which it was dependent upon, and commits those commits were dependent upon, etc etc. I did not see any non-trivial way to do this otherwise. I believe at one point Gergo backported some of his fixes to 7.8, which had since become 'non applicable' (and I thank him for that greatly), but inevitably we instead brought along the few extra changes anyway, since they were *still* needed for other fixes. And some of them had API changes. So the choice was to rewrite 4 patches for an old codebase completely (the work being done by two separate people) or backport a few extra patches. The above is obviously an extreme case. But it stands to reason this would _only happen again_ with 7.8.4, probably even worse since more months of development have gone by. An LTS release would mandate things like no-API-changes-at-all, but this significantly limits our ability to *actually* backport patches sometimes, like the above, due to dependent changes. The alternative, obviously, is to do what Gergo did and manually re-write such a fix for the older branch. But that means we would have had to do that for *every patch* in the same boat, including 2 or 3 other fixes we needed! Furthermore, while I am a release manager and do think I know a bit about GHC, it is hopeless to expect me to know it all. I will absolutely require coordinated effort to help develop 'retropatches' that don't break API compatibility, from active developers who are involved in their respective features. And they are almost all volunteers! Simon and I are the only ones who wouldn't qualify on that. So - at what point does it stop becoming 'backporting fixes to older versions' and instead become literally "working on the older version of the compiler AND the new one in tandem"? Given our rate of churn and change internally, this seems like it would be a significant burden in general to ask of developers. If we had an LTS release of GHC that lasted 3 years for example, that would mean developers are expected to work on the current code of their own, *and their old code for the next three years*. That is an absolutely, undeniably a _huge_ investment to ask of someone. It's not clear how many can actually hold it (and I don't blame them). This email is already a bit long (which is extremely unusual for my emails, I'm sure you all know), but I just wanted to give some insight on the process. I think the technical/automation aspects are the easy part. We could probably fully automate the GHC release process in days, if one or two people worked on it dilligently. The hard part is actually balancing the needs and time of users and developers, which is a complex relationship. On Mon, Oct 6, 2014 at 6:22 PM, John Lato wrote:

On Mon, Oct 6, 2014 at 5:38 PM, Johan Tibell wrote:

...

On Mon, Oct 6, 2014 at 11:28 AM, Herbert Valerio Riedel wrote:

...

On 2014-10-06 at 11:03:19 +0200, p.k.f.holzenspies@utwente.nl wrote:

...

The danger, of course, is that people aren't very enthusiastic about bug-fixing older versions of a compiler, but for language/compiler-uptake, this might actually be a Better Way.

Maybe some of the commercial GHC users might be interested in donating the manpower to maintain older GHC versions. It's mostly a time-consuming QA & auditing process to maintain old GHCs.

What can we do to make that process cheaper? In particular, which are the manual steps in making a new GHC release today?

I would very much like to know this as well. For ghc-7.8.3 there were a number of people volunteering manpower to finish up the release, but to the best of my knowledge those offers weren't taken up, which makes me think that the extra overhead for coordinating more people would outweigh any gains. From the outside, it appears that the process/workflow could use some improvement, perhaps in ways that would make it simpler to divide up the workload.

John L.

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

-- Regards, Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

Mmm... yes, you seem to have some strong points against LTS. People committing to an LTS-version aside from the further development of HEAD seems somewhat unlikely also... I must say, though, significant API-changes with only minor version-bumps have bitten me also. Not sure we should want this. Ph. PS. Maybe long, but not too long, let alone TL;DR. Thanks for the clarity ________________________________________ From: mad.one@gmail.com on behalf of Austin Seipp Sent: 07 October 2014 02:45 To: John Lato Cc: Johan Tibell; Holzenspies, P.K.F. (EWI); ghc-devs@haskell.org; Simon Marlow Subject: Re: Tentative high-level plans for 7.10.1 The steps for making a GHC release are here: https://ghc.haskell.org/trac/ghc/wiki/MakingReleases So, for the record, making a release is not *that* arduous, but it does take time. On average it will take me about 1 day or so to go from absolutely-nothing to release announcement: 1. Bump version, update configure.ac, tag. 2. Build source tarball (this requires 1 build, but can be done very quickly). 3. Make N binary builds for each platform (the most time consuming part, as this requires heavy optimizations in the builds). 4. Upload documentation for all libraries. 5. Update webpage and upload binaries. 6. Send announcement. 7. Upload binaries from other systems later. Herbert has graciously begun taking care of stewarding and uploading the libraries. So, there are a few steps we could introduce to alleviate this process technically in a few ways, but ultimately all of these have to happen, pretty much (regardless of the automation involved). But I don't think this is the real problem. The real problem is that GHC moves forward in terms of implementation extremely, extremely quickly. It is not clear how to reconcile this development pace with something like needing dozens of LTS releases for a stable version. At least, not without a lot of concentrated effort from almost every single developer. A lot of it can be alleviated through social process perhaps, but it's not strictly technical IMO. What do I mean by that? I mean that: - We may introduce a feature in GHC version X.Y - That might have a bug, or other problems. - We may fix it, and in the process, fix up a few other things and refactor HEAD, which will be GHC X.Y+2 eventually. - Repeat steps 2-3 a few times. - Now we want to backport the fixes for that feature in HEAD back to X.Y. - But GHC X.Y has *significantly* diverged from HEAD in that timeframe, because of step 3 being repeated! In other words: we are often so aggressive at refactoring code that the *act* of backporting in and of itself can be complicated, and it gets harder as time goes on - because often the GHC of a year ago is so much different than the GHC of today. As a concrete example of this, let's look at the changes between GHC 7.8.2 and GHC 7.8.3: https://github.com/ghc/ghc/compare/ghc-7.8.2-release...ghc-7.8.3-release There are about ~110 commits between 7.8.2 and 7.8.3. But as the 7.8 branch lived on, backporting fixes became significantly more complex. In fact, I estimate close to 30 of those commits were NOT direct 7.8 requirements - but they were brought in because _actual fixes_ were dependent on them, in non-trivial ways. Take for example f895f33 by Simon PJ, which fixes #9023. The problem with f895f33 is that by the time we fixed the bug in HEAD with that commit, the history had changed significantly from the branch. In order to get f895f33 to plant easily, I had to backport *at least* 12 to 15 other commits, which it was dependent upon, and commits those commits were dependent upon, etc etc. I did not see any non-trivial way to do this otherwise. I believe at one point Gergo backported some of his fixes to 7.8, which had since become 'non applicable' (and I thank him for that greatly), but inevitably we instead brought along the few extra changes anyway, since they were *still* needed for other fixes. And some of them had API changes. So the choice was to rewrite 4 patches for an old codebase completely (the work being done by two separate people) or backport a few extra patches. The above is obviously an extreme case. But it stands to reason this would _only happen again_ with 7.8.4, probably even worse since more months of development have gone by. An LTS release would mandate things like no-API-changes-at-all, but this significantly limits our ability to *actually* backport patches sometimes, like the above, due to dependent changes. The alternative, obviously, is to do what Gergo did and manually re-write such a fix for the older branch. But that means we would have had to do that for *every patch* in the same boat, including 2 or 3 other fixes we needed! Furthermore, while I am a release manager and do think I know a bit about GHC, it is hopeless to expect me to know it all. I will absolutely require coordinated effort to help develop 'retropatches' that don't break API compatibility, from active developers who are involved in their respective features. And they are almost all volunteers! Simon and I are the only ones who wouldn't qualify on that. So - at what point does it stop becoming 'backporting fixes to older versions' and instead become literally "working on the older version of the compiler AND the new one in tandem"? Given our rate of churn and change internally, this seems like it would be a significant burden in general to ask of developers. If we had an LTS release of GHC that lasted 3 years for example, that would mean developers are expected to work on the current code of their own, *and their old code for the next three years*. That is an absolutely, undeniably a _huge_ investment to ask of someone. It's not clear how many can actually hold it (and I don't blame them). This email is already a bit long (which is extremely unusual for my emails, I'm sure you all know), but I just wanted to give some insight on the process. I think the technical/automation aspects are the easy part. We could probably fully automate the GHC release process in days, if one or two people worked on it dilligently. The hard part is actually balancing the needs and time of users and developers, which is a complex relationship. On Mon, Oct 6, 2014 at 6:22 PM, John Lato wrote:

On Mon, Oct 6, 2014 at 5:38 PM, Johan Tibell wrote:

...

On Mon, Oct 6, 2014 at 11:28 AM, Herbert Valerio Riedel wrote:

...

On 2014-10-06 at 11:03:19 +0200, p.k.f.holzenspies@utwente.nl wrote:

...

The danger, of course, is that people aren't very enthusiastic about bug-fixing older versions of a compiler, but for language/compiler-uptake, this might actually be a Better Way.

Maybe some of the commercial GHC users might be interested in donating the manpower to maintain older GHC versions. It's mostly a time-consuming QA & auditing process to maintain old GHCs.

What can we do to make that process cheaper? In particular, which are the manual steps in making a new GHC release today?

I would very much like to know this as well. For ghc-7.8.3 there were a number of people volunteering manpower to finish up the release, but to the best of my knowledge those offers weren't taken up, which makes me think that the extra overhead for coordinating more people would outweigh any gains. From the outside, it appears that the process/workflow could use some improvement, perhaps in ways that would make it simpler to divide up the workload.

John L.

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

-- Regards, Austin Seipp, Haskell Consultant Well-Typed LLP, http://www.well-typed.com/

On 2014-10-06 at 11:50:03 +0200, Malcolm Wallace wrote:

On 6 Oct 2014, at 10:28, Herbert Valerio Riedel wrote:

...

As I'm not totally sure what you mean: Assuming we already had decided years ago to follow LTS-style, given GHC 7.0, 7.2, 7.4, 7.6, 7.8 and the future 7.10; which of those GHC versions would you have been considered a LTS version?

We continue to use 7.2, at least partly because all newer versions of ghc have had significant bugs that affect us. In fact, 7.2.2 also has a show-stopping bug, but we patched it ourselves to create our very own custom ghc-7.2.3 distribution.

I'd like to point out that's kinda ironic, as of *all* the GHC releases, you had to stay on the one major release that was considered a non-proper tech-preview... :-) Cheers, hvr

Is the wiki page up to date? https://ghc.haskell.org/trac/ghc/wiki/PartialTypeSignatures I'd move the Constraint Wildcards bit out to an appendix or delete altogether -- it's a distraction since it's not part of the design. Named wildcard are described as a "fourth form" but actually it's "third" I suppose. And aren't they just a vairant of "type wildcards"? You can't name "extra-constraint wildcards" can you? There's a long section on "partial expressions and pattern signatures" but I think the conclusion is "we don't do this". Again, move to an appendix of not-implemented ideas. Try to focus on the actual design. Thanks! Simon ________________________________________ From: ghc-devs [ghc-devs-bounces@haskell.org] on behalf of Thomas Winant [Thomas.Winant@cs.kuleuven.be] Sent: 07 October 2014 17:07 To: ghc-devs@haskell.org Subject: Re: Tentative high-level plans for 7.10.1 Hi, On 2014-10-03 23:35, Austin Seipp wrote:

.. Here are the major patches on Phabricator still needing review, that I think we'd like to see for 7.10.1:

- D168: Partial type signatures ..

As Austin said, our patch implementing Partial Type Signatures is still up for code review on Phabricator [1]. It is our goal too to get it in 7.10.1, and we will try to do as much as we can to help out with this process. We'd like it very much if people had a thorough look at it (thanks Richard for the feedback). We're glad to provide additional info (including extra comments in the code), rewrite confusing code, etc. = Status = The implementation is nearly complete: * We've integrated support for Holes, i.e. by default, an underscore in a type signature will generate an error message mentioning the inferred type. By enabling -XPartialTypeSignatures, the inferred type is used and the underscore can remain in the type signature. * SPJ's proposed simplifications (over Skype) have been implemented, except for the fact that we still use the annotated constraints for solving, see [2]. * Richard's comments on Phabricator [1] have been addressed in extra commits. * I've rebased the patch against master on Monday. * I've added docstring for most of the new functions I've added. * Some TODOs still remain, I'll summarise the most important ones here. See [3] for a detailed list with examples. * When -XMonoLocalBinds is enabled (implied by -XGADTs and -XTypeFamilies), (some) local bindings without type signature aren't generalised. Partial type signatures should follow this behaviour. This is currently not handled correctly. We have a stopgap solution involving generating an error in mind, but would prefer a real fix. We'd like some help with this. * Partial type signatures are currently ignored for pattern bindings. This bug doesn't seem to be difficult to solve, but requires some debugging. * The following code doesn't type check: {-# LANGUAGE MonomorphismRestriction, PartialTypeSignatures #-} charlie :: _ => a charlie = 3 Type error: No instance for (Num a) arising from the literal ‘3’. We would like the (Num a) constraint to be inferred (because of the extra-constraint wildcard). * Some smaller things, e.g. improving error messages. We'll try to fix the remaining TODOs, but help is certainly appreciated and will speed up integrating this patch! Please have a look at the code and let us know what we can do to help. Cheers, Thomas Winant [1]: https://phabricator.haskell.org/D168 [2]: https://ghc.haskell.org/trac/ghc/wiki/PartialTypeSignatures#extra-constraint... [3]: https://ghc.haskell.org/trac/ghc/wiki/PartialTypeSignatures#TODOs Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs