gitlab.haskell.org maintenance 2025-01-10

Hi folks, In about 24 hours, I will perform some maintenance on gitlab.haskell.org. I will be addressing the frequent disk outages by improving how partitions are used. This will only require a system restart, so you can expect some short downtime.

On Thu, 9 Jan 2025 at 09:54, Bryan Richter
Hi folks,
In about 24 hours, I will perform some maintenance on gitlab.haskell.org. I will be addressing the frequent disk outages by improving how partitions are used. This will only require a system restart, so you can expect some short downtime.

Well, everything went according to plan, but the plan had a flaw. ;)
GitLab is back up, but the maintenance did not succeed. I will try again in
an hour.
On Fri, 10 Jan 2025 at 09:48, Bryan Richter
On Thu, 9 Jan 2025 at 09:54, Bryan Richter
wrote: Hi folks,
In about 24 hours, I will perform some maintenance on gitlab.haskell.org. I will be addressing the frequent disk outages by improving how partitions are used. This will only require a system restart, so you can expect some short downtime.

Hi, Thanks Bryan for taking on the upgrade. Is it possible to perform a rollback to the last known good state for this afternoon so that we can make some progress with work? Cheers, Matt On Fri, Jan 10, 2025 at 8:44 AM Bryan Richter via ghc-devs < ghc-devs@haskell.org> wrote:
Well, everything went according to plan, but the plan had a flaw. ;)
GitLab is back up, but the maintenance did not succeed. I will try again in an hour.
On Fri, 10 Jan 2025 at 09:48, Bryan Richter
wrote: On Thu, 9 Jan 2025 at 09:54, Bryan Richter
wrote: Hi folks,
In about 24 hours, I will perform some maintenance on gitlab.haskell.org. I will be addressing the frequent disk outages by improving how partitions are used. This will only require a system restart, so you can expect some short downtime.
_______________________________________________
ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Sadly, no. The issue is at a level below backups. And for a reason I
haven't looked into yet, there are no previous NixOS generations available
to roll back to.
The principal issue is that while booting, NixOS Stage 1 prints the message
"cannot import 'rpool': more than one matching pool". The root filesystem
is in that pool, so booting fails.
Unfortunately I don't know why it says that. From within a rescue system, I
can see just one pool named "rpool", and I can import it and mount
filesystems from it just fine. So the current task is figuring out why
NixOS Stage 1 sees otherwise.
On Fri, 10 Jan 2025 at 14:04, Matthew Pickering
Hi,
Thanks Bryan for taking on the upgrade.
Is it possible to perform a rollback to the last known good state for this afternoon so that we can make some progress with work?
Cheers,
Matt
On Fri, Jan 10, 2025 at 8:44 AM Bryan Richter via ghc-devs < ghc-devs@haskell.org> wrote:
Well, everything went according to plan, but the plan had a flaw. ;)
GitLab is back up, but the maintenance did not succeed. I will try again in an hour.
On Fri, 10 Jan 2025 at 09:48, Bryan Richter
wrote: On Thu, 9 Jan 2025 at 09:54, Bryan Richter
wrote: Hi folks,
In about 24 hours, I will perform some maintenance on gitlab.haskell.org. I will be addressing the frequent disk outages by improving how partitions are used. This will only require a system restart, so you can expect some short downtime.
_______________________________________________
ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

It's back!
Ancient, bad ZFS metadata was causing an issue during boot. I eventually
found where the metadata was lurking and was able to remove it. I can
speculate why this wasn't a problem in earlier boots, but I haven't
actually looked into it. (The last boot was 2 years ago!)
The problem had nothing to do with recent changes made by me or anyone
else. It may be related to operating system upgrades that have happened
over the last two years. Changes to the OS may have exposed the lurking
filesystem issue.
As a bonus, my *completely unrelated* filesystem changes have taken effect,
and I believe we'll see a lot fewer disk outages going forward.
-Bryan
On Fri, 10 Jan 2025 at 15:13, Bryan Richter
Sadly, no. The issue is at a level below backups. And for a reason I haven't looked into yet, there are no previous NixOS generations available to roll back to.
The principal issue is that while booting, NixOS Stage 1 prints the message "cannot import 'rpool': more than one matching pool". The root filesystem is in that pool, so booting fails.
Unfortunately I don't know why it says that. From within a rescue system, I can see just one pool named "rpool", and I can import it and mount filesystems from it just fine. So the current task is figuring out why NixOS Stage 1 sees otherwise.
On Fri, 10 Jan 2025 at 14:04, Matthew Pickering < matthewtpickering@gmail.com> wrote:
Hi,
Thanks Bryan for taking on the upgrade.
Is it possible to perform a rollback to the last known good state for this afternoon so that we can make some progress with work?
Cheers,
Matt
On Fri, Jan 10, 2025 at 8:44 AM Bryan Richter via ghc-devs < ghc-devs@haskell.org> wrote:
Well, everything went according to plan, but the plan had a flaw. ;)
GitLab is back up, but the maintenance did not succeed. I will try again in an hour.
On Fri, 10 Jan 2025 at 09:48, Bryan Richter
wrote: On Thu, 9 Jan 2025 at 09:54, Bryan Richter
wrote: Hi folks,
In about 24 hours, I will perform some maintenance on gitlab.haskell.org. I will be addressing the frequent disk outages by improving how partitions are used. This will only require a system restart, so you can expect some short downtime.
_______________________________________________
ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
participants (2)
-
Bryan Richter
-
Matthew Pickering