On Fri, Aug 08, 2014 at 09:00:21AM +0200, Herbert Valerio Riedel wrote:

Just to clarify, as the last sentence contains a double-negation: GHC devs continue pushing to github.com/haddock.git's `master` branch to keep Haddock building with GHC HEAD? It's just that the Haddock development proper happens in a branch other than `master` from now on?

From my perspective I would prefer to use `master` for Haddock development and use a branch with some other name for GHC development. My main motivation here is that as a contributor to Haddock "I expect the latest code to be on `master`, and I would use it as a base when developing new features".

Alternatively, maybe use `master` for both Haddock and GHC development, but push to different remotes (say use http://git.haskell.org/haddock.git for GHC development and https://github.com/haskell/haddock for Haddock development). I think this is what we already do for e.g. `containers`. Cheers, Simon

On Fri, Aug 08, 2014 at 10:35:44AM +0200, Herbert Valerio Riedel wrote:

If this is just an alpha-conversion modulo thing, then let's just call the new branch for GHC HEAD simply `ghc-head` (or something like that) and keep hosting it in github.com/haskell/haddock.git, and have GHC HEAD developers push to that instead (fwiw, you can specify the default branch in .gitmodules, which some few Git tools honor).

Ok, cool, that would work for me. Cheers.

On Fri, Aug 8, 2014 at 10:11 AM, Simon Peyton Jones wrote:

The biggest disadvantage in my mind is that you're setting yourself up for a potentially huge merge just before the GHC release and might block the GHC release until that merge is done (assuming that haddock is still shipped with GHC).

Excellent point.

The merge shouldn’t block the release, though. In extremis, I guess we could always release the GHC fork of Haddock if the tip of Haddock wasn’t merged to match GHC! But I doubt it’ll come to that

But as you mentioned the GHC fork of Haddock might not work (it might just type check) so at the very least Mateusz is signing up for validating that it indeed works before a GHC release. That's of course fine, I just want people to understand what we're signing up for.

On Fri, Aug 8, 2014 at 5:13 PM, Mateusz Kowalczyk wrote:

...

On Fri, Aug 8, 2014 at 10:11 AM, Simon Peyton Jones < simonpj@microsoft.com> wrote:

...

The biggest disadvantage in my mind is that you're setting yourself up for a potentially huge merge just before the GHC release and might block the GHC release until that merge is done (assuming that haddock is still shipped with GHC).

Excellent point.

The merge shouldn’t block the release, though. In extremis, I guess we could always release the GHC fork of Haddock if the tip of Haddock wasn’t merged to match GHC! But I doubt it’ll come to that

But as you mentioned the GHC fork of Haddock might not work (it might just type check) so at the very least Mateusz is signing up for validating

On 08/08/2014 10:15 AM, Johan Tibell wrote: that

...

it indeed works before a GHC release. That's of course fine, I just want people to understand what we're signing up for.

Well, I stick around and am usually aware of GHC release early. In the usual case Haddock will be fixed up before the actual GHC release. I don't think API changes were ever drastic enough to provide major problems especially seeing as I'll be able to refer to the GHC-tracked branch to see what patches were applied there.

However let's consider I can't make it for the release because I'm not available around that time or otherwise. This should still not hold up GHC release. I would expect the GHC team to release Haddock + their fixes, it would simply be like an existing release with some patches on top to have it work with new GHC. I can then come around and once I apply any API patches, I make an actual Haddock release. People can then simply cabal install haddock and use what they get here rather than what came with GHC.

Be careful hear so 1) patches aren't lost (that almost happened once when GHC HQ made a containers release) and 2) that the version numbers used by GHC HQ and your releases make sense (i.e. follow the PVP). P.S. I would recommend naming the main development branch 'master' and the other 'ghc-head'. 'master' is the branch new potential developers will see first on GitHub and it's the one people default to when making pull requests. I used to have the main 'network' development in a branch called 'develop'. This led to lots of confusion and pull requests against the wrong branches. The name 'ghc-head' also makes it much clearer what that branch is for and why it might be special.

I'm also in favour of a more decoupled development/release process. I'd like to change a few things in haddock to make it more suitable for use as a library, so that I can set up a haddock for GHCJS without duplicating the whole package (it needs a custom platform setup and some changes in file name handling). It'd be great if such a release could be made independently of GHC and changes like this could be made without requiring the user to update their GHC. Also I'd be happy to do some of the work for the out-of-tree change, for example backporting fixes to the 2.14 branch or updating the 2.15 branch to work with the 7.8 api (but I can't promise more, GHCJS is taking enough of my time, and I'm not sure how much time I can afford to keep spending on it, so I'd like to minimize my other maintenance tasks as much as possible). luite On Fri, Aug 8, 2014 at 10:18 AM, Simon Marlow wrote:

I thought this was what you were already doing :-) Anyway, this is more or less the setup we had in mind when Haddock was added to the GHC tree. The only question is which branches are used for GHC and for regular development, and where they live. As long as that's clear for everyone (both Haddock and GHC developers), then this should be fine.

The GHC release engineer will need to give the Haddock maintainers plenty of heads-up time before a release so that the merge can be done - Austin could you add that to the release checklist?

Cheers, Simon

On 08/08/2014 06:25, Mateusz Kowalczyk wrote:

...

Hello,

A slightly long e-mail but I ask that you voice your opinion if you ever changed GHC API. You can skim over the details, simply know that it saves me vast amount of time, allows me to try and find contributors and doesn't impact GHC negatively. It seems like a win-win scenario for GHC and Haddock. GHC team's workflow does not change and will not require any new commitment: I do all the work and I think it's a 1 line change in sync-all when transition is ready. Here it is:

It is no secret that many core Haskell projects lack developer hands and Haddock is no exception: the current maintainers are Simon Hengel and myself. Simon does not have much time so currently all the issues and updates are up to me. Ideally I would like if some more people could come and hack on Haddock but there are a couple of problems with trying to recruit folk for this:

1. Interacting with GHC API is not the easiest thing. This is Haddock's problem but I thought I'd mention it here.

2. Haddock resides directly in the GHC tree and it is currently *required* that it compiles with GHC HEAD. This is a huge barrier of entry for anyone: today I wanted to make a fairly simple change but it still took me 3 validate runs to be at least somewhat confident that I didn't break much in GHC. On top of this I had help from Edward Z. Yang on IRC and information from him on what the issue exactly was. If I was to do everything alone it would have taken even more validates. A validate is not fast on machine by any means, it takes an hour or two.

Here is what I want to do unless there are major objections: I want to move the active development away from GHC tree. Below is how it would work. For simplicity please imagine that we have *just* released 7.8.3.

* Haddock development would concentrate on supporting the last public release of GHC: I stop developing against GHC HEAD and currently would develop against 7.8.3.

* GHC itself checks out Haddock as a submodule as it does now. The only difference is that it points at whatever commit worked last. Let us assume it is the Haddock 2.14.3 release commit. The vital difference from current state is that GHC will no longer track changes in master branch.

* Now when GHC API changes things proceed as they normally do: whoever is responsible for the changes, pops into the Haddock submodule applies the patches necessary for Haddock to build with HEAD and everyone is happy. What does *not* happen is these patches don't go into master: I ignore them and keep working with 7.8.3.

* When a GHC release rolls around, I update Haddock to work with the new API so that people with new release can still use it. Once it works against new API, GHC can start tracking from that commit onwards and proceed as usual.

Here are the advantages:

* I don't have to work against GHC HEAD. This means I don't have to build GHC HEAD and I don't need to worry about GHC API changes. I don't waste 2-4 hours building before hacking and validating after hacking to make any minor changes and to make sure I haven't broken anything.

* More importantly, anyone who wants to write a patch for Haddock can now do so easily, all they need is recent compiler rather than being forced to build HEAD. Building and validating against HEAD is a **huge** barrier of entry.

* I only have to care about GHC API changes once a release and not twice a week. I think PatternSynonyms have changed 4 times in a month but the end result at release time is the same and that's what people care about.

* It is less work for anyone changing GHC API: they only have to deal with their own changes and not my changes which add features or whatever.

* If I break something in Haddock HEAD, GHC is not affected.

* If Haddock's binary interface doesn't change, we may even allow more versions of GHC be compatible through CPP and other such trickery. If we were to do it today, it would be an increased burden on the GHC team to deal with those.

* I can release as often as I want against the same compiler version. Currently doing this requires backporting features (see v2.14 branch) which is a massive pain. I no longer have to tell the users ‘yes, your bug is fixed but to get it you need to compile GHC HEAD or wait 6-12 months until next GHC release’. I have to do this a lot.

Here are the disadvantages and why I think they don't make a big difference:

* GHC HEAD doesn't get any new-and-cool features that we might implement. I say this doesn't matter because no one uses varying GHC HEAD versions to develop actual software, documentation and all. What I mean to say is that the only user of the Haddock that's developed in GHC tree is GHC itself. The only case where GHC actually used in-tree Haddock was when Herbert generated documentation for base-4.7 early for me to eye before the release. Even this doesn't matter because so close to the release I'll already have the existing GHC API integrated anyway. Again, it does not matter if GHC HEAD itself doesn't get pretty operator rendering or whatever right when I implement it because no one cares about it until it's release time. I know that many people simply HADDOCK_DOCS=NO to save time. The actual users only care about Haddock that works with 7.6.x, 7.8.x, 7.10.x; only GHC cares about in-betweens and only for the purpose of being able to build and validate.

* GHC team can't easily contribute features and get the back immediately. In part it doesn't matter because of the previous point and in the last year or so there were no features contributed directly from GHC except those necessary to keep Haddock compiling. This just means there's no demand for such close relationship.

* Haddock-affecting changes in GHC parser don't ‘take effect’ straight away. This is my loss and considering the infrequency at which such changes happen, it's a tiny price to pay to have to wait until release.

* …that's it, no other disadvantages that I can think of, but that's why I'm sending it to the list to review!

What's worth mentioning is that the no-external-dependencies thing still applies because even though we no longer need to compile against HEAD, we still need to compile against the tree at release time.

In summary:

My life gets easier because I stop wasting it on playing with whole GHC tree, GHC team's life gets easier because they don't have to deal with the changes I make. My life gets even easier because I only have to make big API updates once a release. I can actually start looking for contributors.

When a release rolls around, GHC and Haddock ‘meet up’, we make sure it all works, release happens, GHC starts tracking from that point and we part ways until the next release.

What do you think? If there are no major objections in one week then I will assume I am good to go with this.

Transition from current setup: If I receive some patches I was promised then I will then make a 2.14.4 bugfix/compat release make sure that master is up to date and then create something like GHC-tracking branch from master and track that. I will then abandon that branch and not push to it unless it is GHC release time. The next commit in master will bring Haddock to a state where it works with 7.8.3: yes, this means removing all new API stuff until 7.10 or 7.8.4 or whatever. GHC API changes go onto GHC-tracking while all the stuff I write goes master. When GHC makes a release or is about to, I make master work with that and make GHC-tracking point to that instead.

Thanks!

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

On 08/08/2014 10:18 AM, Simon Marlow wrote:

I thought this was what you were already doing :-) Anyway, this is more or less the setup we had in mind when Haddock was added to the GHC tree. The only question is which branches are used for GHC and for regular development, and where they live. As long as that's clear for everyone (both Haddock and GHC developers), then this should be fine.

I think there is no problem if they both live in the existing repository (github.com/haskell/haddock) or whatever the submodule refers to today.

The GHC release engineer will need to give the Haddock maintainers plenty of heads-up time before a release so that the merge can be done - Austin could you add that to the release checklist?

Right, although I don't exactly plan to abandon any of the GHC information channels I'm on today: I tend to be well aware of a release coming.

Cheers, Simon

-- Mateusz K.

On 2014-08-14 at 00:09:40 +0200, Mateusz Kowalczyk wrote: [...]

I don't know what the GHC branch name will be yet. ‘ghc-head’ makes most sense but IIRC Herbert had some objections as it had been used in the past for something else, but maybe he can pitch in.

I had no objections at all to that name, 'ghc-head' is fine with me :-)

In an ideal world, all GHC developers would also think about how to add Haddock support for the wonderful features they are adding, and code them up themselves. In practice, Haddock support has never stopped a feature from getting into GHC, but I think people who do add features should also be willing to roll up their sleeves and help the Haddock folks support them, though maybe at a later point in time... Edward Excerpts from Carter Schonwald's message of 2014-08-14 01:43:50 +0100:

one thing I wonder about is how should we approach noting "theres a new language constructor, we should figure out a good way to present it in haddock" in this work flow? because the initial haddocks presentation might just be a strawman till someone thinks about it carefully right?

On Wed, Aug 13, 2014 at 6:30 PM, Herbert Valerio Riedel wrote:

...

On 2014-08-14 at 00:09:40 +0200, Mateusz Kowalczyk wrote:

[...]

...

I don't know what the GHC branch name will be yet. ‘ghc-head’ makes most sense but IIRC Herbert had some objections as it had been used in the past for something else, but maybe he can pitch in.

I had no objections at all to that name, 'ghc-head' is fine with me :-) _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

good points by all :) On Thu, Aug 14, 2014 at 12:48 PM, Mateusz Kowalczyk

wrote:

On 08/14/2014 01:43 AM, Carter Schonwald wrote:

...

one thing I wonder about is how should we approach noting "theres a new language constructor, we should figure out a good way to present it in haddock" in this work flow? because the initial haddocks presentation might just be a strawman till someone thinks about it carefully right?

On Wed, Aug 13, 2014 at 6:30 PM, Herbert Valerio Riedel < hvriedel@gmail.com> wrote:

...

On 2014-08-14 at 00:09:40 +0200, Mateusz Kowalczyk wrote:

[...]

...

I don't know what the GHC branch name will be yet. ‘ghc-head’ makes most sense but IIRC Herbert had some objections as it had been used in the past for something else, but maybe he can pitch in.

I had no objections at all to that name, 'ghc-head' is fine with me :-) _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

If there's more than one reasonable way then there's going to be strawman along the way somewhere anyway but we can at least delegate that until later. As I mention in the OP, there's at least no need for me to worry about it until it's finished on the GHC side although I'll no doubt be aware of it sooner than that.

The PatternSynonyms stuff is an example where the implementor also stepped up to putting in support into Haddock for rendering. At the same time, the implementation has changed multiple times along the way creating hassle for both parties so perhaps in the future it's better to simply make sure Haddock still compiles and works but perhaps delegate everything else to closer to the release.

In the end, it does not matter if Haddock can't display a bleeding edge feature until it's going out as a release.

-- Mateusz K. _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs

Great. Please can what you do be documented clearly somewhere, with a link to that documentation from here https://ghc.haskell.org/trac/ghc/wiki/Repositories, and/or somewhere else suitable? Thanks Simon | -----Original Message----- | From: ghc-devs [mailto:ghc-devs-bounces@haskell.org] On Behalf Of | Mateusz Kowalczyk | Sent: 13 August 2014 23:10 | To: ghc-devs@haskell.org | Subject: Re: Moving Haddock *development* out of GHC tree | | On 08/08/2014 06:25 AM, Mateusz Kowalczyk wrote: | > Hello, | > | > [snip] | > | > Transition from current setup: | > If I receive some patches I was promised then I will then make a | > 2.14.4 bugfix/compat release make sure that master is up to date and | > then create something like GHC-tracking branch from master and track | > that. I will then abandon that branch and not push to it unless it is | > GHC release time. The next commit in master will bring Haddock to a | > state where it works with 7.8.3: yes, this means removing all new API | > stuff until 7.10 or 7.8.4 or whatever. GHC API changes go onto | > GHC-tracking while all the stuff I write goes master. When GHC makes | a | > release or is about to, I make master work with that and make | > GHC-tracking point to that instead. | > | > | > Thanks! | > | | So it is now close to a week gone and I have received many positive | replies and no negative ones. I will probably execute what I stated | initially at about this time tomorrow. | | To reiterate in short: | | 1. I make sure what we have now compiles with GHC HEAD and I stick it | in separate branch which GHC folk will now track and apply any API | patches to. Unless something changes by tomorrow, this will most likely | be what master is at right now, perhaps with a single change to the | version in cabal file. | | 2. I make the master branch work with 7.8.3 (and possibly 7.8.x) and do | development without worrying about any API changes in HEAD, releasing | as often as I need to. | | 3. At GHC release time, I update master with API changes so that up-to- | date Haddock is ready to be used to generate the docs and ship with the | compiler. | | I don't know what the GHC branch name will be yet. 'ghc-head' makes | most sense but IIRC Herbert had some objections as it had been used in | the past for something else, but maybe he can pitch in. | | The only thing I require from GHC folk is to simply use that branch and | not push/pull to/from master unless contributing feature patches or | trying to port some fixes into HEAD version for whatever reason. | | Thanks! | | -- | Mateusz K. | _______________________________________________ | ghc-devs mailing list | ghc-devs@haskell.org | http://www.haskell.org/mailman/listinfo/ghc-devs

On 2014-08-16 at 16:59:51 +0200, Mateusz Kowalczyk wrote: [...]

Herbert kindly updated the sync-all script that defaults to the new branch so I think we're covered.

Minor correction: I did not touch the sync-all script at all. I merely declared a default branch in the .gitmodules file: http://git.haskell.org/ghc.git/commitdiff/03a8003e5d3aec97b3a14b2d3c774aad43...

On 08/25/2014 01:21 PM, Alan & Kim Zimmerman wrote:

What happens in the case of a change to the dev branch of ghc that requires a patch to haddock as well, how does that patch get added to phabricator, or is there a separate process?

A case in point is https://phabricator.haskell.org/D157 with matching change at https://github.com/alanz/haddock/tree/wip/landmine-param-family

Regards Alan

You need to push the patch against the Haddock ghc-head branch and update the submodule reference to point at your patch. I don't think that you need to do anything special for Phabricator unless it does some weird checking out instead of using whatever references GHC points to. -- Mateusz K.

Hello Herbert, I think the pre-commit hook needs to be adjusted; I used to have push rights on master, but I cannot seem to push to ghc-head. Thanks, Edward Excerpts from Herbert Valerio Riedel's message of 2014-08-16 16:34:46 +0100:

On 2014-08-16 at 16:59:51 +0200, Mateusz Kowalczyk wrote:

[...]

...

Herbert kindly updated the sync-all script that defaults to the new branch so I think we're covered.

Minor correction: I did not touch the sync-all script at all. I merely declared a default branch in the .gitmodules file:

http://git.haskell.org/ghc.git/commitdiff/03a8003e5d3aec97b3a14b2d3c774aad43...