The disk issue is fixed now, but one day later Gitlab is broken again. This time
I'm getting random 502s.
I tired to check logs, I don't know what the problem is but I saw one error line
which may be relevant
Aug 11 12:21:09 gitlab.haskell.org unicorn[3605]:
/nix/store/h6ppx34ccb3binw7awbphaicv5q938z5-ruby2.5.5-prometheus-client-mmap-0.9.8/lib/ruby/gems/2.5.0/gems/prometheus-client-mmap-0.9.8/lib/prometheus/client/mmaped_dict.rb:42:
[BUG] Bus Error at 0x00007fe3cd6f6000
Ömer
Ömer Sinan Ağacan
Hi,
Just yesterday Gitlab was giving 500 because the disk was full. Ben deleted some files, but in less than 24h it's full again. This started happening regularly, I wonder if we could do something about this.
The reason this time seems to be that Gitlab started generating 22G-large backups daily since the 7th. I'm not sure how important those backups are so I'm not deleting them.
There's also a large docker-registry directory (101G).
I think it might be good to set up some kind of downtime monitoring or maybe something on the Gitlab server to send an email when the disk is nearly full. It could send an email to people who has access to the server.
It'd also be good to come up with an action plan when this happens. I have access to the server, but I have no idea which files are important. Documenting Gitlab setup (and the server details) in more details might be helpful.
Does anyone have any other ideas to keep the server running?
Ömer