Hi Ben, Thanks for getting the fix in so quickly. However: suppose you were unavailable to do this, due to a well-deserved holiday perhaps. Who else has the credentials and know-how to step in? I feel as if we should have a resource somewhere with a definitive list of critical services and who has access to what. Perhaps this list should be kept private, in case the knowledge itself is a (small) security risk -- but it should be written down and shared widely enough that it is unlikely for all people with access to be unavailable at the same time. Do we have such a resource already? Richard

On Mar 19, 2021, at 9:47 AM, Ben Gamari wrote:

Simon Peyton Jones via ghc-devs writes:

...

GHC's GitLab seems to be down. Ben? (I just get 502's)

Everything should now be back to normal. I have sent a post-mortem describing the failure mode and fix. Apologies for the inconvenience!

Cheers,

- Ben _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

There is the haskell-infrastructure group, who has access. Its knowledge of the peculiarities of each of the very different services that's harder to keep up to date between people. If Ben was hit by a bus it is prepared to be handled. With this diversity and level of manpower though stepping in tends to be a high cost operation so it is done when necessary. -davean On Fri, Mar 19, 2021 at 10:00 AM Richard Eisenberg wrote:

Hi Ben,

Thanks for getting the fix in so quickly.

However: suppose you were unavailable to do this, due to a well-deserved holiday perhaps. Who else has the credentials and know-how to step in?

I feel as if we should have a resource somewhere with a definitive list of critical services and who has access to what. Perhaps this list should be kept private, in case the knowledge itself is a (small) security risk -- but it should be written down and shared widely enough that it is unlikely for all people with access to be unavailable at the same time.

Do we have such a resource already?

Richard

...

On Mar 19, 2021, at 9:47 AM, Ben Gamari wrote:

Simon Peyton Jones via ghc-devs writes:

...

GHC's GitLab seems to be down. Ben? (I just get 502's)

Everything should now be back to normal. I have sent a post-mortem describing the failure mode and fix. Apologies for the inconvenience!

Cheers,

- Ben _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

On Mar 19, 2021, at 12:44 PM, howard.b.golden@gmail.com wrote:

I would like to help however I can. I already maintain the Haskell wiki, and I would like to improve and document its configuration using devops techniques, preferably consistent with gitlab.haskell.org http://gitlab.haskell.org/.

Thanks, Howard! I will try to take you up on your offer to help: do you think you could start this documentation process more broadly? That is, not just covering the Haskell Wiki, but also, say, gitlab.haskell.org. (You say you wish to document the wiki's configuration consistently with gitlab.haskell.org, but I don't know that the latter is documented!) Ideally, I would love to know what services haskell.org hosts, who runs them, and what happens if those people become unavailable. There's a zoo of services out there, and knowing who does what would be invaluable. Of course, anyone can start this process, but it takes someone willing to stick with it and see it through for a few weeks. Since Howard boldly stepped forward, I nominate him. :) Thanks, Richard

Cc: admin@haskell.org which remains (since it was set up over five? years ago) the contact address for haskell infra admin stuff. There's also, as there has always been, the #haskell-infrastructure irc channel. This all _used_ to be on phabricator (and was on the haskell wiki before that) but it wasn't really suited to port over nicely to gitlab. At the request of the still-forming working group for HF, a repo was created on github with key information and policy, and sent to the HF, which should have this documented somewhere already: https://github.com/haskell-infra/haskell-admins I'll go back and try to read earlier in the thread, if I was cc'd, because I'm just now looped in? I.e. I see from the message "gitlab is down" but other than that I'm not sure exactly what is at issue? -g --- Here's the info I sent the committee last time it asked: here's the full list of servers. benley (Benjamin Staffin) has been using the two boxes as his tests for mailman migration. the various servers tagged bgamari are all the ghc specific ones. the three hackage servers are matrix, docbuilder, and hackage proper. the two consolidated servers are misc-services-origin and www-combo-origin. here's info on packet's elastic block storage: https://www.packet.com/developers/docs/storage/ebs/ its a bit fiddly, but we have it working at this point. Davean, Herbert, Ben and me are the ones who have keys on almost every box, I think. Alp and Austin are both officially team members but I haven't interacted with them much (austin is legacy, and alp may only work with Ben on ghc stuff, idk). Davean manages backups. some significant subdomains hosted: archives.haskell.org, hoogle.haskell.org, downloads.haskell.org, wiki.haskell.org there are some other smaller subdomains such as summer.haskell.org, pvp.haskell.org, etc. here are the main static subsites of haskell.org, most of which have existed at those urls well prior to the existence of a haskell.org website beyond the wiki alex communities ghc-perf happy hoogle onlinereport arrows definition ghcup haskell-symposium hugs platform cabal ghc haddock haskell-workshop nhc98 tutorial Sadly, joachim didn't keep up ghc-perf which was nice while it was working. On Fri, Mar 19, 2021 at 2:13 PM wrote:

Hi Richard, Gershom and Ben,

I have access to the server that runs the Haskell wiki. There are other websites on that server as well. I can document them as well. I know that Gershom B. has done most (all?) of the work on that server. I ask him and the haskell.org committee to send me or point me at whatever documentation they have already.

I ask Ben and others with any documentation of gitlab.haskell.org to send me or point me at whatever they have already.

I don't want to do this alone. Other volunteers are welcome! I also will abide by the Haskell Foundation and the haskell.org committee in arranging this documentation according to their needs and preferences.

Howard

On Fri, 2021-03-19 at 17:32 +0000, Richard Eisenberg wrote:

...

...

On Mar 19, 2021, at 12:44 PM, howard.b.golden@gmail.com wrote:

I would like to help however I can. I already maintain the Haskell wiki, and I would like to improve and document its configuration using devops techniques, preferably consistent with gitlab.haskell.org.

Thanks, Howard!

I will try to take you up on your offer to help: do you think you could start this documentation process more broadly? That is, not just covering the Haskell Wiki, but also, say, gitlab.haskell.org. (You say you wish to document the wiki's configuration consistently with gitlab.haskell.org, but I don't know that the latter is documented!)

Ideally, I would love to know what services haskell.org hosts, who runs them, and what happens if those people become unavailable. There's a zoo of services out there, and knowing who does what would be invaluable.

Of course, anyone can start this process, but it takes someone willing to stick with it and see it through for a few weeks. Since Howard boldly stepped forward, I nominate him. :)

Thanks, Richard

Thanks, Gershom, for the quick and very illuminating response.

On Mar 19, 2021, at 2:21 PM, Gershom B wrote:

Cc: admin@haskell.org which remains (since it was set up over five? years ago) the contact address for haskell infra admin stuff.

How would I learn of that address? Who is admin@haskell.org?

There's also, as there has always been, the #haskell-infrastructure irc channel.

How would I learn of this?

This all _used_ to be on phabricator (and was on the haskell wiki before that) but it wasn't really suited to port over nicely to gitlab. At the request of the still-forming working group for HF, a repo was created on github with key information and policy, and sent to the HF, which should have this documented somewhere already: https://github.com/haskell-infra/haskell-admins

This is exactly the kind of stuff I was looking for. (Apologies for your need to repeat yourself -- we are still working hard to organize ourselves out of the void.) How would I find this page? Perhaps just a link from the Committees section of https://www.haskell.org/community/ would be sufficient. Might also be good to put admin@haskell.org in the footer.

I'll go back and try to read earlier in the thread, if I was cc'd, because I'm just now looped in?

I.e. I see from the message "gitlab is down" but other than that I'm not sure exactly what is at issue?

The thread morphed into a new topic, thanks to me. I was just wondering who I would reach out to if both GitLab and Ben were down. The haskell-admins repo answers that question nicely.

-g

---

Davean, Herbert, Ben and me are the ones who have keys on almost every box, I think. Alp and Austin are both officially team members but I haven't interacted with them much (austin is legacy, and alp may only work with Ben on ghc stuff, idk).

This seems out of sync with https://github.com/haskell-infra/haskell-admins, which does not list Ben (or Austin or Alp) but does list Rick Elrod. I would think that dormant members (e.g. Austin) should be removed, but that's your call.

Davean manages backups.

Is this documented somewhere? Backups of what?

some significant subdomains hosted: archives.haskell.org, hoogle.haskell.org, downloads.haskell.org, wiki.haskell.org

there are some other smaller subdomains such as summer.haskell.org, pvp.haskell.org, etc.

here are the main static subsites of haskell.org, most of which have existed at those urls well prior to the existence of a haskell.org website beyond the wiki

alex communities ghc-perf happy hoogle onlinereport

arrows definition ghcup haskell-symposium hugs platform

cabal ghc haddock haskell-workshop nhc98 tutorial

This stuff appears to be on https://github.com/haskell-infra/haskell-admins/blob/master/servers.md. Good. I was originally looking for each of these to have names associated with them, but perhaps if one needed to contact the owner of a particular subdomain, we could reach out to admin@haskell.org and then get the subdomain owners by dereference. That's fine by me.

Sadly, joachim didn't keep up ghc-perf which was nice while it was working.

I think this is perf.haskell.org. To be fair, some of the failure belongs on my shoulders. I offered up an unused server at Bryn Mawr (my previous employer) to host this, but that situation proved to be flaky. Actually, my best guess is that the server is still actually alive (and in the basement of my old building) at perf.haskell.org, but not being updated. I don't think we should rely on it. So: Howard, there may not be much work to do here, beyond emailing Gershom, as you've already done! :) Richard

On Fri, Mar 19, 2021 at 2:13 PM wrote:

...

Hi Richard, Gershom and Ben,

I have access to the server that runs the Haskell wiki. There are other websites on that server as well. I can document them as well. I know that Gershom B. has done most (all?) of the work on that server. I ask him and the haskell.org committee to send me or point me at whatever documentation they have already.

I ask Ben and others with any documentation of gitlab.haskell.org to send me or point me at whatever they have already.

I don't want to do this alone. Other volunteers are welcome! I also will abide by the Haskell Foundation and the haskell.org committee in arranging this documentation according to their needs and preferences.

Howard

On Fri, 2021-03-19 at 17:32 +0000, Richard Eisenberg wrote:

...

...

On Mar 19, 2021, at 12:44 PM, howard.b.golden@gmail.com wrote:

I would like to help however I can. I already maintain the Haskell wiki, and I would like to improve and document its configuration using devops techniques, preferably consistent with gitlab.haskell.org.

Thanks, Howard!

I will try to take you up on your offer to help: do you think you could start this documentation process more broadly? That is, not just covering the Haskell Wiki, but also, say, gitlab.haskell.org. (You say you wish to document the wiki's configuration consistently with gitlab.haskell.org, but I don't know that the latter is documented!)

Ideally, I would love to know what services haskell.org hosts, who runs them, and what happens if those people become unavailable. There's a zoo of services out there, and knowing who does what would be invaluable.

Of course, anyone can start this process, but it takes someone willing to stick with it and see it through for a few weeks. Since Howard boldly stepped forward, I nominate him. :)

Thanks, Richard

On Fri, Mar 19, 2021 at 07:08:35PM +0000, Richard Eisenberg wrote:

...

On Mar 19, 2021, at 2:21 PM, Gershom B wrote: Cc: admin@haskell.org which remains (since it was set up over five? years ago) the contact address for haskell infra admin stuff.

How would I learn of that address? Who is admin@haskell.org?

...

There's also, as there has always been, the #haskell-infrastructure irc channel.

How would I learn of this?

By way of clarification, both of these contact details are mentioned in the document that Gershom linked: https://github.com/haskell-infra/haskell-admins#the-team-and-how-to-contact-... Thanks for that document, Gershom, it's very helpful! Tom

I can try to step up and be backup on the other side of the planet. Ben and I are almost 12hs apart exactly. On Sat, 20 Mar 2021 at 1:32 AM, Richard Eisenberg wrote:

On Mar 19, 2021, at 12:44 PM, howard.b.golden@gmail.com wrote:

I would like to help however I can. I already maintain the Haskell wiki, and I would like to improve and document its configuration using devops techniques, preferably consistent with gitlab.haskell.org.

Thanks, Howard!

I will try to take you up on your offer to help: do you think you could start this documentation process more broadly? That is, not just covering the Haskell Wiki, but also, say, gitlab.haskell.org. (You say you wish to document the wiki's configuration consistently with gitlab.haskell.org, but I don't know that the latter is documented!)

Ideally, I would love to know what services haskell.org hosts, who runs them, and what happens if those people become unavailable. There's a zoo of services out there, and knowing who does what would be invaluable.

Of course, anyone can start this process, but it takes someone willing to stick with it and see it through for a few weeks. Since Howard boldly stepped forward, I nominate him. :)

Thanks, Richard _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Just a heads up everyone. Gitlab appears down again. This seemed to have happened around Sunday, 4AM UTC. Everyone have a blissful Sunday! On Sat, 20 Mar 2021 at 6:05 PM, Giorgio Marinelli wrote:

I can also help (~UTC+1), I've a long history and experience in systems management and engineering.

Best,

Giorgio Marinelli https://marinelli.dev/cv

On Sat, 20 Mar 2021 at 00:31, Moritz Angermann wrote:

...

I can try to step up and be backup on the other side of the planet. Ben

and I are almost 12hs apart exactly.

...

On Sat, 20 Mar 2021 at 1:32 AM, Richard Eisenberg

...

...

On Mar 19, 2021, at 12:44 PM, howard.b.golden@gmail.com wrote:

I would like to help however I can. I already maintain the Haskell wiki, and I would like to improve and document its configuration using devops techniques, preferably consistent with gitlab.haskell.org.

Thanks, Howard!

I will try to take you up on your offer to help: do you think you could

start this documentation process more broadly? That is, not just covering

wrote: the Haskell Wiki, but also, say, gitlab.haskell.org. (You say you wish to document the wiki's configuration consistently with gitlab.haskell.org, but I don't know that the latter is documented!)

...

...

Ideally, I would love to know what services haskell.org hosts, who

runs them, and what happens if those people become unavailable. There's a zoo of services out there, and knowing who does what would be invaluable.

...

Of course, anyone can start this process, but it takes someone willing

to stick with it and see it through for a few weeks. Since Howard boldly stepped forward, I nominate him. :)

...

Thanks, Richard _______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

It appears as if gitlab went down again, this time since around 4AM UTC on Monday. On Sun, Mar 21, 2021 at 9:49 PM Ben Gamari wrote:

Moritz Angermann writes:

...

Just a heads up everyone. Gitlab appears down again. This seemed to have happened around Sunday, 4AM UTC.

Everyone have a blissful Sunday!

It is back up, again. It appears that GitLab's backup retention logic is now either broken or has changed since it is now failing to delete old backups, resulting a full disk. I'll be keeping an eye on this until we sort out the root cause.

Cheers,

- Ben