Why don't we just put Haddock into GHC's repository? It was proposed in a previous discussion in February [1] and it would avoid the bad experience of having it as a submodule while keeping it in sync. With the following commands we can keep the whole commit history: In Haddock repo:

mkdir -p utils/haddock git rm .arcconfig .arclint .ghci .gitignore .travis.yml git mv -k * utils/haddock git commit -a -m "Prepare Haddock merge"

In GHC repo:

git rm -rf utils/haddock git commit -a -m "Prepare Haddock merge" git remote add haddock https://gitlab.haskell.org/ghc/haddock git fetch haddock git merge --allow-unrelated-histories haddock/ghc-8.6 -m "Merge haddock" git remote remove haddock

[1] https://mail.haskell.org/pipermail/ghc-devs/2019-February/017120.html Cheers, Sylvain On 06/03/2019 13:08, Ben Gamari wrote:

Ryan Scott writes:

...

I do think something is afoot here. The current Haddock submodule commit is at 07f2ca [1], but the ghc-head branch of Haddock is still at commit 8459c6 [2]. It would be good if someone could update the ghc-head branch accordingly.

Indeed. Done.

It would be nice if we had a better way to handle this. Ideally Marge or someone similar would land any relevant haddock patches to ghc-head when landing a GHC MR.

Cheers,

- Ben

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

If the status quo is poor, then why not merge the two? git-subtree won't fix the problem and will arguably be even harder to manage than submodules. Best, On Wed, 6 Mar 2019 at 15:59, Ben Gamari wrote:

Sylvain Henry writes:

...

Why don't we just put Haddock into GHC's repository? It was proposed in a previous discussion in February [1] and it would avoid the bad experience of having it as a submodule while keeping it in sync.

I'm reluctant to keep it in the GHC tree; it really is a separate project with separate maintainership, dependencies, and contributors.

That being said, I do agree that the status quo is pretty poor. I was going to suggest managing it with git-subtree instead, but I'm not sure this would be much of an improvement.

Cheers,

- Ben _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Merging Haddock into GHC’s subtree would undoubtedly simplify the GHC workflow for patches affecting Haddock. There are a couple other considerations though: Haddock currently sees the bulk of its contributions and bug fixes on its stable branches, not the ghc-head branch (which is the one GHC tracks). Not only is this exactly the opposite of GHC, but it also makes outside contributions and CI much simpler. The Haddock project is actually 3 packages, two of which are libraries, one of which is expected to build on GHC going all the way back to the 7.4.* series. AFAICT, the packages kept in GHC’s source tree usually advance in lockstep with GHC. Running the full GHC CI for most of Haddock’s changes seems a waste of resources and an unnecessary delay. Besides, Haddock actually has a larger test suite, which is only expected to compile on the stable branch (although I try to keep it running on ghc-head with the help of head.hackage). I recently incorporated the other part of Haddock’s testsuite into GHC’s testsuite, so I’m hoping there will be less breakage coming from ghc-head. Interleaving git histories would (mostly) just make bisecting both GHC and Haddock more of a pain. The right way to solve this problem is probably to find a better way of factoring GHC-specific functionality out and putting only that in the GHC tree. This is a good long term goal, but I don’t think we are quite there yet. Some other ongoing changes in both GHC and Haddock are blocking the way forward on this front… Thanks, Alec

On Mar 6, 2019, at 1:26 PM, Boespflug, Mathieu wrote:

If the status quo is poor, then why not merge the two? git-subtree won't fix the problem and will arguably be even harder to manage than submodules.

Best,

On Wed, 6 Mar 2019 at 15:59, Ben Gamari mailto:ben@smart-cactus.org> wrote: Sylvain Henry mailto:sylvain@haskus.fr> writes:

...

Why don't we just put Haddock into GHC's repository? It was proposed in a previous discussion in February [1] and it would avoid the bad experience of having it as a submodule while keeping it in sync.

I'm reluctant to keep it in the GHC tree; it really is a separate project with separate maintainership, dependencies, and contributors.

That being said, I do agree that the status quo is pretty poor. I was going to suggest managing it with git-subtree instead, but I'm not sure this would be much of an improvement.

Cheers,

- Ben _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org mailto:ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

I tried adding back the linters which check to make sure a commit is in the upstream branch before a MR is merged but got blocked by a gitlab issue. https://gitlab.haskell.org/ghc/ghc/merge_requests/395 The currently recommended workflow is that your commit should be in the ghc-head branch before the merge to GHC takes place. This enforces some linearisation but it stops the tree breaking. Matt On Wed, Mar 13, 2019 at 9:17 AM Spiwack, Arnaud wrote:

On Wed, Mar 6, 2019 at 11:04 PM Alec Theriault wrote:

...

The right way to solve this problem is probably to find a better way of factoring GHC-specific functionality out and putting only that in the GHC tree. This is a good long term goal, but I don’t think we are quite there yet. Some other ongoing changes in both GHC and Haddock are blocking the way forward on this front…

In the meantime, there is no way to make atomic updates to GHC and Haddock (which need to happen regularly). And GHC's master and the ghc-head branch keep getting out of sync. It's really hard to diagnose, until it blocks someone's valuable time. At which point it's too late.

Is there a short term solution which would alleviate that cost, besides merging Haddock in the main Ghc tree? _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Under your workflow it doesn't seem like the commit which ends up in master will point to a commit on ghc-head? This is problematic when doing bisection if the branch where the commit lives is deleted or force pushed to. I would prefer a workflow which is more annoying for contributors but doesn't leave the tree in a bad state than one which is convenient but dangerous. Cheers, Matt On Wed, Mar 13, 2019 at 1:42 PM Alec Theriault wrote:

Hi,

The currently recommended workflow is that your commit should be in the ghc-head branch before the merge to GHC takes place. This enforces

This seems problematic: everyone is going to race to get their changes into Haddock's ghc-head first, then block everyone else’s Haddock-touching patches from building with CI until the GHC side of the first person's changes goes through too. And that might take some time if Marge finds problems with the patch. I propose the workflow be:

Once you’ve finished your patch, rebase the GHC side on top of upstream GHC master Rebase the Haddock side on top of upstream Haddock ghc-head Once Marge merges your MR, fast-forward the upstream ghc-head to your new commit

That way, multiple MR’s with Haddock parts can “race” to get merged. Whoever loses just has to rebase. Ideally, we would have Marge doing both the rebasing and step #3 for us. In the place of Marge, anyone who has write access to Haddock should feel free to do step #3 too (like when Marge merges an MR while the author of the MR is busy doing other things… like sleeping).

As a side note, I do wish there was an easy way in GitLab to quickly jump to the diffs of submodules (or at least an easy way to copy the new commit hash).

Thanks, Alec

On Mar 13, 2019, at 4:13 AM, Matthew Pickering wrote:

I tried adding back the linters which check to make sure a commit is in the upstream branch before a MR is merged but got blocked by a gitlab issue.

https://gitlab.haskell.org/ghc/ghc/merge_requests/395

The currently recommended workflow is that your commit should be in the ghc-head branch before the merge to GHC takes place. This enforces some linearisation but it stops the tree breaking.

Matt

On Wed, Mar 13, 2019 at 9:17 AM Spiwack, Arnaud wrote:

On Wed, Mar 6, 2019 at 11:04 PM Alec Theriault wrote:

The right way to solve this problem is probably to find a better way of factoring GHC-specific functionality out and putting only that in the GHC tree. This is a good long term goal, but I don’t think we are quite there yet. Some other ongoing changes in both GHC and Haddock are blocking the way forward on this front…

In the meantime, there is no way to make atomic updates to GHC and Haddock (which need to happen regularly). And GHC's master and the ghc-head branch keep getting out of sync. It's really hard to diagnose, until it blocks someone's valuable time. At which point it's too late.

Is there a short term solution which would alleviate that cost, besides merging Haddock in the main Ghc tree? _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

What is this bad state? I think the problem is that if you don’t push to haddock/ghc-head first, then the commit in ghc/master would point to a commit which has no guarantee to be durable: it may be rebased or squashed, by the time it gets to ghc-head. This means that after the commit has been mutated, the commit in ghc is no longer a valid checkout (because it can’t checkout its Haddock submodule). In the current situation, it is imperative that haddock/ghc-head is updated first. If we want to preserve the most minimal of invariant. It is still a pretty awful situation to be in, to be honest. - It’s rather hard to enforce, so if one forgets to update haddock/ghc-head first, then we are back to square one. (I suppose that, in constrast, in Matthew’s linter change, we would have red CI during the entire review phase of an MR. Which would be rather inconvenient, to say the least) - You need two independent sets of approval to get either part of a patch merged. - Two MR which want to modify Haddock will needlessly conflict with one another. When one has its Haddock modification merged. The other requires updates (it’s super tedious to rebase a history and be clean if there are several commit which touch Haddock, by the way). And trigger unnecessary consumption of our already precious CI resources. - In fact, any merging of a change to haddock/ghc-head will require rewriting the history of the MR that needs it. Also unnecessary consumption of CI resources. Quite frankly, this is not a fun place to be. It may be worth taking a step back and contemplating the situation we’ve built for ourselves. /Arnaud On Wed, Mar 13, 2019 at 3:19 PM Alec Theriault wrote:

Under your workflow it doesn't seem like the commit which ends up in master will point to a commit on ghc-head?

Correct, but at most the head commit will be out of sync (and a fast forward merge should be always possible). Step 3 is about rectifying that situation.

This is problematic when doing bisection if the branch where the commit lives is deleted or force pushed to.

We still end up with an easy-to-bisect linear history (in both Haddock and GHC) at the end of the day. I fear I may be misunderstanding your point here.

I would prefer a workflow which is more annoying for contributors but doesn't leave the tree in a bad state than one which is convenient but dangerous.

What is this bad state? Perhaps I’m again misunderstanding, but how does your proposal help leave the tree in a better state? As soon as someone preparing a GHC patch pushes changes directly to ghc-head and before those changes get merged, the situation is going to be just as confusing for everyone else trying to do something with Haddock (and not easily fixable either).

Thanks, Alec

On Mar 13, 2019, at 6:59 AM, Matthew Pickering < matthewtpickering@gmail.com> wrote:

Under your workflow it doesn't seem like the commit which ends up in master will point to a commit on ghc-head?

This is problematic when doing bisection if the branch where the commit lives is deleted or force pushed to.

I would prefer a workflow which is more annoying for contributors but doesn't leave the tree in a bad state than one which is convenient but dangerous.

Cheers,

Matt

On Wed, Mar 13, 2019 at 1:42 PM Alec Theriault wrote:

Hi,

The currently recommended workflow is that your commit should be in the ghc-head branch before the merge to GHC takes place. This enforces

This seems problematic: everyone is going to race to get their changes into Haddock's ghc-head first, then block everyone else’s Haddock-touching patches from building with CI until the GHC side of the first person's changes goes through too. And that might take some time if Marge finds problems with the patch. I propose the workflow be:

Once you’ve finished your patch, rebase the GHC side on top of upstream GHC master Rebase the Haddock side on top of upstream Haddock ghc-head Once Marge merges your MR, fast-forward the upstream ghc-head to your new commit

That way, multiple MR’s with Haddock parts can “race” to get merged. Whoever loses just has to rebase. Ideally, we would have Marge doing both the rebasing and step #3 for us. In the place of Marge, anyone who has write access to Haddock should feel free to do step #3 too (like when Marge merges an MR while the author of the MR is busy doing other things… like sleeping).

As a side note, I do wish there was an easy way in GitLab to quickly jump to the diffs of submodules (or at least an easy way to copy the new commit hash).

Thanks, Alec

On Mar 13, 2019, at 4:13 AM, Matthew Pickering < matthewtpickering@gmail.com> wrote:

I tried adding back the linters which check to make sure a commit is in the upstream branch before a MR is merged but got blocked by a gitlab issue.

https://gitlab.haskell.org/ghc/ghc/merge_requests/395

The currently recommended workflow is that your commit should be in the ghc-head branch before the merge to GHC takes place. This enforces some linearisation but it stops the tree breaking.

Matt

On Wed, Mar 13, 2019 at 9:17 AM Spiwack, Arnaud wrote:

On Wed, Mar 6, 2019 at 11:04 PM Alec Theriault wrote:

The right way to solve this problem is probably to find a better way of factoring GHC-specific functionality out and putting only that in the GHC tree. This is a good long term goal, but I don’t think we are quite there yet. Some other ongoing changes in both GHC and Haddock are blocking the way forward on this front…

In the meantime, there is no way to make atomic updates to GHC and Haddock (which need to happen regularly). And GHC's master and the ghc-head branch keep getting out of sync. It's really hard to diagnose, until it blocks someone's valuable time. At which point it's too late.

Is there a short term solution which would alleviate that cost, besides merging Haddock in the main Ghc tree? _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs