Action Required: Status of Harbormaster for Differentials

newer
RE: [commit: ghc] master: Remove...

older
Trying to refactor DeriveAnyClass,...

Ben Gamari

26 Aug 2016 26 Aug '16

9:38 a.m.

tl;dr. I am working on restoring Harbormaster support for Differentials. Towards this end, GHC's Phabricator instance will soon require that you add an SSH public key to your account in order to upload patches with `arc diff`. Hello everyone, As you are no doubt aware Harbormaster has not been building Differentials for a very long time now. I am currently looking at options for restoring this service but it appears that doing so will require a bit of action on the part of our contributors. Specically, you will need to add an SSH key to Phabricator. You can do so by navigating to the Phabricator Settings [1] page, selecting the "Personal Account Settings" option, and selecting "SSH Public Keys" from the list on the left. There you can click on the "SSH Key Actions" menu on the top-right corner and click on the "Upload Public Key" item. Here you can specify a key name and paste the key contents (e.g. the contents of $HOME/.ssh/id_rsa.pub). The reason for this new requirement is that `arc diff` will soon push submitted diffs to a "staging area", a Git repository managed by Phabricator containing a branch for each submitted Differential. This will allow more reliable Harbormaster builds and merges as we will no longer need to rely on Phabricator correctly applying Differential patches (which has historically been problematic, especially for Differentials where the base commit is unavailable). If there are no objections, I would like to enable the staging area for the rGHC repository in three days, on Monday, 29 August 2016. After this date `arc diff` will try to push submitted differentials to the staging area, which will require a public key. Note, however, that pushing to the staging area can always be disabled with `arc diff`'s `--skip-staging` flag (although we won't be able to run CI builds on Differentials submitted in this way). To be clear: I would have liked to provide an option that did not require key-based authentication, but sadly neither Phabricator nor Gitolite provide any such option. I apologize for the added friction that this new requirement imposes. If enough people feel strongly that this is too onerous then I'm happy to entertain alternative solutions. Cheers, - Ben [1] https://phabricator.haskell.org/settings/

Attachments:

signature.asc (application/pgp-signature — 472 bytes)

Show replies by date

Joachim Breitner

26 Aug 26 Aug

10:56 a.m.

Hi, Am Freitag, den 26.08.2016, 09:38 -0400 schrieb Ben Gamari:

...

The reason for this new requirement is that `arc diff` will soon push submitted diffs to a "staging area", a Git repository managed by Phabricator containing a branch for each submitted Differential.

this is great, because it means that we can run performance builds via perf.haskell.org on all DRs automatically! Will the staging repository be publicly available? And will it also contain the usual branches (master, ghc-8.0, wip/), so that I can simply take that as the upstream repo for gipeda’s sake? Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • https://www.joachim-breitner.de/ XMPP: nomeata@joachim-breitner.de • OpenPGP-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org

Ben Gamari

5:39 p.m.

Joachim Breitner writes:

...

[ Unknown signature status ] Hi,

Am Freitag, den 26.08.2016, 09:38 -0400 schrieb Ben Gamari:

...
The reason for this new requirement is that `arc diff` will soon push submitted diffs to a "staging area", a Git repository managed by Phabricator containing a branch for each submitted Differential.

this is great, because it means that we can run performance builds via perf.haskell.org on all DRs automatically!

Will the staging repository be publicly available?

Indeed, it's available at [1]. There are no

...

And will it also contain the usual branches (master, ghc-8.0, wip/), so that I can simply take that as the upstream repo for gipeda’s sake?

Not as things are currently configured and I'm not sure how easy it would be to reliably accomplish this. The only thing you'll find in that repository currently is tags (not branches as I stated earlier; sorry!) with names of the form phabricator/base/1234 and phabricator/diff/1234. These contain the base and head commits of revision 1234, respectively. Note that 1234 does not identify a differential; rather it is a revision of a differential (e.g. the number that you see in the "ID" column of the history table fn a differential in the web interface). Cheers, - Ben [1] https://phabricator.haskell.org/diffusion/GHCDIFF/

Phyx

27 Aug 27 Aug

12:20 p.m.

Hi Ben, Out of curiosity which platforms will be getting harbormaster support? Is this only Linux x86_64? Any plans to expand this in the future to all Tier 1 platforms? Cheers, Tamar On Fri, Aug 26, 2016, 22:39 Ben Gamari wrote:

...

Joachim Breitner writes:

...
[ Unknown signature status ] Hi,

Am Freitag, den 26.08.2016, 09:38 -0400 schrieb Ben Gamari:

...
The reason for this new requirement is that `arc diff` will soon push submitted diffs to a "staging area", a Git repository managed by Phabricator containing a branch for each submitted Differential.

this is great, because it means that we can run performance builds via perf.haskell.org on all DRs automatically!

Will the staging repository be publicly available?

Indeed, it's available at [1]. There are no

...
And will it also contain the usual branches (master, ghc-8.0, wip/), so that I can simply take that as the upstream repo for gipeda’s sake?

Not as things are currently configured and I'm not sure how easy it would be to reliably accomplish this.

The only thing you'll find in that repository currently is tags (not branches as I stated earlier; sorry!) with names of the form phabricator/base/1234 and phabricator/diff/1234. These contain the base and head commits of revision 1234, respectively. Note that 1234 does not identify a differential; rather it is a revision of a differential (e.g. the number that you see in the "ID" column of the history table fn a differential in the web interface).

Cheers,

- Ben

[1] https://phabricator.haskell.org/diffusion/GHCDIFF/ _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Ben Gamari

29 Aug 29 Aug

9:04 a.m.

Phyx writes:

...

Hi Ben,

Out of curiosity which platforms will be getting harbormaster support? Is this only Linux x86_64?

Any plans to expand this in the future to all Tier 1 platforms?

Indeed I would like to get to this point. At the moment I am focusing on Linux x86_64 and Darwin x86_64. I may try to bring up an ARM as well. The other platforms would be subject to machine availability. Cheers, - Ben

Joachim Breitner

28 Aug 28 Aug

8:29 a.m.

Hi, Am Freitag, den 26.08.2016, 17:39 -0400 schrieb Ben Gamari:

...

Indeed, it's available at [1].

it does not seem to be world-readable: $ git fetch phab fatal: unable to access 'http://phabricator.haskell.org/diffusion/GHCDIFF/GHC-Differentials.git/': The requested URL returned error: 500

...

The only thing you'll find in that repository currently is tags (not branches as I stated earlier; sorry!) with names of the form phabricator/base/1234 and phabricator/diff/1234. These contain the base and head commits of revision 1234, respectively. Note that 1234 does not identify a differential; rather it is a revision of a differential (e.g. the number that you see in the "ID" column of the history table fn a

...
ifferential in the web interface).

Ok, I think I will have to add custom code to take care of DRs in gipeda, but it is probably worth it. Can I assume that phabricator/base/1234 is always (or at least usually) a commit on master, or some other branch in the ghc repository? Greetings, Joachim -- Joachim “nomeata” Breitner mail@joachim-breitner.de • https://www.joachim-breitner.de/ XMPP: nomeata@joachim-breitner.de • OpenPGP-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org

Ben Gamari

29 Aug 29 Aug

10:03 a.m.

Joachim Breitner writes:

...

[ Unknown signature status ] Hi,

Am Freitag, den 26.08.2016, 17:39 -0400 schrieb Ben Gamari:

...
Indeed, it's available at [1].

it does not seem to be world-readable:

$ git fetch phab fatal: unable to access 'http://phabricator.haskell.org/diffusion/GHCDIFF/GHC-Differentials.git/': The requested URL returned error: 500

Actually it looks like this was a configuration issue which broke Phabricator's HTTP git hosting. I believe I have fixed this but now seem to be fighting our CDN which I'll need help from the infrastructure guys for. Regardless, you should be able to clone the repo via ssh.

...

...
The only thing you'll find in that repository currently is tags (not branches as I stated earlier; sorry!) with names of the form phabricator/base/1234 and phabricator/diff/1234. These contain the base and head commits of revision 1234, respectively. Note that 1234 does not identify a differential; rather it is a revision of a differential (e.g. the number that you see in the "ID" column of the history table fn a

...
ifferential in the web interface).

Ok, I think I will have to add custom code to take care of DRs in gipeda, but it is probably worth it.

This code won't strictly speaking be in gipeda, right? Rather, it would merely be a bit more logic in the build script, no?

...

Can I assume that phabricator/base/1234 is always (or at least usually) a commit on master, or some other branch in the ghc repository?

Often it will be. However, I have in the past often opened "chains" of Differentials where the HEAD of one is the base commit of the next. Naturally in this case the base commit will not be in master. Cheers, - Ben

Joachim Breitner

10:19 a.m.

Hi, Am Montag, den 29.08.2016, 10:03 -0400 schrieb Ben Gamari:

...

...
it does not seem to be world-readable:

$ git fetch phab fatal: unable to access 'http://phabricator.haskell.org/diffusion/G HCDIFF/GHC-Differentials.git/': The requested URL returned error: 500

Actually it looks like this was a configuration issue which broke Phabricator's HTTP git hosting. I believe I have fixed this but now seem to be fighting our CDN which I'll need help from the infrastructure guys for.

Regardless, you should be able to clone the repo via ssh.

Permission denied (publickey,keyboard-interactive). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. (This is without using an ssh key added to Phab – after all, I need a build service to poll that, and don’t want to hand out keys to that.) I’ll wait for the CDN to get working.

...

...
...
The only thing you'll find in that repository currently is tags (not branches as I stated earlier; sorry!) with names of the form phabricator/base/1234 and phabricator/diff/1234. These contain the base and head commits of revision 1234, respectively. Note that 1234 does not identify a differential; rather it is a revision of a differential (e.g. the number that you see in the "ID" column of the history table fn a

...
ifferential in the web interface).

Ok, I think I will have to add custom code to take care of DRs in gipeda, but it is probably worth it.

This code won't strictly speaking be in gipeda, right? Rather, it would merely be a bit more logic in the build script, no?

Both. The build script needs to know that there are two repositories to poll from Gipeda needs to produce a section, like the one for branches, for DRs, which have slightly different semantics (in particular, they have an explicit base commit that comparisons should be made against, which is not the case for git branches). Hmm. Links to git diffs would also have to go to a different place. Annoying complexity there. Maybe a separate gipeda instance is simpler (but would get in the way of comparing commits on Phab with commits on the main repo, and duplicate building work). I’ll have to think more about how this can be done in a viable way. Greetings, Joachim -- -- Joachim “nomeata” Breitner mail@joachim-breitner.de • https://www.joachim-breitner.de/ XMPP: nomeata@joachim-breitner.de • OpenPGP-Key: 0xF0FBF51F Debian Developer: nomeata@debian.org

Ben Gamari

10:29 a.m.

Joachim Breitner writes:

...

Hi,

Am Montag, den 29.08.2016, 10:03 -0400 schrieb Ben Gamari:

...
Regardless, you should be able to clone the repo via ssh.

Permission denied (publickey,keyboard-interactive). fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

(This is without using an ssh key added to Phab – after all, I need a build service to poll that, and don’t want to hand out keys to that.)

Ahh, sure. Right, it's expected that ssh requires a key. Indeed http is more fitting for your bot.

...

I’ll wait for the CDN to get working.

I'll let you know when things have been sorted.

...

...
...
Ok, I think I will have to add custom code to take care of DRs in gipeda, but it is probably worth it.

This code won't strictly speaking be in gipeda, right? Rather, it would merely be a bit more logic in the build script, no?

Both.

The build script needs to know that there are two repositories to poll from

Gipeda needs to produce a section, like the one for branches, for DRs, which have slightly different semantics (in particular, they have an explicit base commit that comparisons should be made against, which is not the case for git branches).

Ahhh, I see. A nice point. Cheers, - Ben

Ben Gamari

11:17 a.m.

Ben Gamari writes:

...

Joachim Breitner writes:

...
Hi,

...

...

Ahh, sure. Right, it's expected that ssh requires a key. Indeed http is more fitting for your bot.

...
I’ll wait for the CDN to get working.

I'll let you know when things have been sorted.

Well this is unfortunate: it looks like GHC might be too big [1] for Phabricator's HTTP git hosting logic. Unfortunately it's hard to claim this with much confidence since the endpoint provides no error message beyond a 500 status code. I'm not really sure what to advise here; I'll try to raise a flag with the Phacility people but even if I could confirm that T4369 is the culprit, it's a bug that still unfixed after more than two years and I don't hold out much hope that there will be a fix in the near-term. They suggest that affected users instead use SSH. Sadly this would mean that you would need to provide a key to your bot. It is certainly not a very satisfying answer, but I'm afraid it is likely the best we'll be able to do for a while. Cheers, - Ben [1] https://secure.phabricator.com/T4369

Ben Gamari

3:33 p.m.

Ben Gamari writes:

...

tl;dr. I am working on restoring Harbormaster support for Differentials. Towards this end, GHC's Phabricator instance will soon require that you add an SSH public key to your account in order to upload patches with `arc diff`.

For the record, I just flipped the switch on this and enabled building of Differentials with Harbormaster. So far things look good but please let me know if you encounter any problems. Thanks! Cheers, - Ben

Richard Eisenberg

30 Aug 30 Aug

10:38 a.m.

Hi Ben, Thanks for tackling this difficult issue -- I know you've been looking for a better solution. As I understand it, anyone will now need to add a public key to Phab in order to upload patches. This (as you admit) will increase the barriers to contributions. Are the instructions for how a new contributor should submit patches (without a public key and therefore without CI) posted very visibly somewhere? Thanks, Richard

...

On Aug 26, 2016, at 9:38 AM, Ben Gamari wrote:

tl;dr. I am working on restoring Harbormaster support for Differentials. Towards this end, GHC's Phabricator instance will soon require that you add an SSH public key to your account in order to upload patches with `arc diff`.

Hello everyone,

As you are no doubt aware Harbormaster has not been building Differentials for a very long time now. I am currently looking at options for restoring this service but it appears that doing so will require a bit of action on the part of our contributors.

Specically, you will need to add an SSH key to Phabricator. You can do so by navigating to the Phabricator Settings [1] page, selecting the "Personal Account Settings" option, and selecting "SSH Public Keys" from the list on the left. There you can click on the "SSH Key Actions" menu on the top-right corner and click on the "Upload Public Key" item. Here you can specify a key name and paste the key contents (e.g. the contents of $HOME/.ssh/id_rsa.pub).

The reason for this new requirement is that `arc diff` will soon push submitted diffs to a "staging area", a Git repository managed by Phabricator containing a branch for each submitted Differential. This will allow more reliable Harbormaster builds and merges as we will no longer need to rely on Phabricator correctly applying Differential patches (which has historically been problematic, especially for Differentials where the base commit is unavailable).

If there are no objections, I would like to enable the staging area for the rGHC repository in three days, on Monday, 29 August 2016. After this date `arc diff` will try to push submitted differentials to the staging area, which will require a public key. Note, however, that pushing to the staging area can always be disabled with `arc diff`'s `--skip-staging` flag (although we won't be able to run CI builds on Differentials submitted in this way).

To be clear: I would have liked to provide an option that did not require key-based authentication, but sadly neither Phabricator nor Gitolite provide any such option. I apologize for the added friction that this new requirement imposes. If enough people feel strongly that this is too onerous then I'm happy to entertain alternative solutions.

Cheers,

- Ben

[1] https://phabricator.haskell.org/settings/ _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Ben Gamari

4:04 p.m.

Richard Eisenberg writes:

...

Hi Ben,

Thanks for tackling this difficult issue -- I know you've been looking for a better solution.

As I understand it, anyone will now need to add a public key to Phab in order to upload patches. This (as you admit) will increase the barriers to contributions. Are the instructions for how a new contributor should submit patches (without a public key and therefore without CI) posted very visibly somewhere?

I have some edits of the Phabricator page [1] in the Wiki on-going as we speak (although mpickering deserves credit for doing the original edits working in mentions of the SSH key requirement). Cheers, - Ben [1] https://ghc.haskell.org/trac/ghc/wiki/Phabricator

3186

Age (days ago)

3190

Last active (days ago)

List overview

Download

12 comments

5 participants

participants (5)

Ben Gamari
Ben Gamari
Joachim Breitner
Phyx
Richard Eisenberg