I eventually resorted to a server reboot, which cleared up all the problems I was seeing. I think we're back in business.

Symptoms were:

* No new data coming into https://grafana.gitlab.haskell.org/d/iiCppweMz/marge-bot?orgId=2&from=now-7d&to=now&refresh=30m

* High-frequency repetition of the system log message "systemd-journald[1622008]: Failed to open runtime journal: Device or resource busy"

* ~50% failure rate connecting to the server with ssh

None of those are happening anymore.

-Bryan

On Mon, 20 Mar 2023 at 14:41, Bryan Richter <bryan@haskell.foundation> wrote:

I am seeing a few different problems with the GitLab server right now. I am gonna try to mitigate the issues, so the server might be unavailable for a few short periods.