Hello GHC devs, The gitlab should be fine again, it looks like the spammers have left on their own. Here is my analysis of the problem. If you have an idea on what to do or are against my proposed "solution", please say so. 1. We had a much higher load than usual, leading to congestion on the workers. 2. No matter what, we could not have added new workers, since either they would get reaped via gitlabs normal worker reaping or they would be OOMkilled when the entire application exceeded available RAM 3. Gitlab's own reverse proxy (yes, we have two) talks to the web workers via a unix domain socket. 4. When we hit a certain load, the workers could not take requests from the queue quickly enough 5. instead of stopping, the reverse proxy kept hammering the the queue 6. When the queue didn't fill up, you would just get a lot of client cancels after a couple seconds 7. When the queue *did* fill up, the gitlab would start throwing "service unavailable" errors *and immediately return with a 502* 8. Now a vicious cycle begins: the clients (the spammers) would *immediately* resubmit their request instead of timing out, which kept the queue congested. The only fix is to restart the gitlab, which of course clears up the queue until it becomes congested again, from which point on it kept being congested. (which is why I had a couple of situations where I went "oh I fixed it" but then after half an hour and a spike of traffic later, we were back to 502s) The conclusion here is much sadder though: 1. I did an analysis of the origin and locked out a couple of bad /24 subnets but they only made up a tiny fraction of the traffic -- overall, I could not identify "big bad subnets" 2. delegated blocks came from random (as far as cloud providers are random given their are only three of them, we could e.g. see AWS) cloud providers. This is expected, as bad actors of course wouldn't have their own infra which would be pretty easy to ban wholesale 3. This is why IP address based blocking (i.e. rate limiting) did not work, which we're doing on the gitlab level (per IP) and on the reverse proxy level (for /24s) (our nginx, not gitlab's own reverse proxy). 4. The only thing that would really help is more workers, which also isn't a great fix. They could just hammer with more requests *and* the workers are taking up an absurd amount of RAM (currently about 50GB, every worker is 1-2 more GB, so convervatively one worker is 2 GB of ram extra that we need) 5. Or to reintroduce anubis or some tarpit on the expensive routes. Maybe this is one of the easiest fixes. We would of course only have this for unauthenticated requests and on expensive routes. The worst offender being graphql requests. Best Magnus On 4/20/26 15:15, Andreas Klebinger via ghc-devs wrote:
Hello Devs,
just a short update and some background about Gitlab issues today.
Over the week and in particular today the number of requests to our GitLab instance has grown steadily, it now receiving way more than doubled the number of requests it received last week.
We have tried to solve this by adjusting some GitLab/server settings to deal with this (rate limits, checking for spammy IPs, the works) but with little success so far since it seems that the requests are pretty distributed across their origins.
Given it's not garbage requests but for valid endpoints we assume it's "just" crawlers (likely driven by the AI hype) indexing our instance. But while this isn't a DoS attack in it's totality the effect similar and it's unclear when we will make progress on improving the situation.
Magnus is working towards resolving this issue and we will let you know.
Cheers Andreas
On 20/04/2026 10:52, Magnus Viernickel via ghc-devs wrote:
Hello GHC devs,
we're seeing a lot of 502s at the moment, we're getting twice the rate of requests than normal, I am investigating what is going on and trying to fix this.
Best
Magnus
_______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org
ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org