I'm afraid I'm going to repeat myself, but I fail to see the value of GHC developers becoming experts in infrastructure management and DDoS defense mitigation. It's time to stop wasting resources on this and acknowledge that self-hosting a forge has been a humongous failure and has caused great damage to the GHC project, its image and the contribution experience. I'm grateful for Magnus' efforts, but I think it's time that the GHC project is making hard decisions. There are ways to be less dependent on forges... or rely on hosted services, because they have better infrastructure and manpower. On August 6, 2026 7:16:41 PM UTC, Simon Peyton Jones via ghc-devs <ghc-devs@haskell.org> wrote:
Thank you Magnus for your heroic efforts. It sounds like a frustrating task.
And yet, isn't there a simple solution?
1. Provide a read/write interface to GitLab for logged-in authenticated users only 2. Also provide a read-only Gitlab, feeding off the same database (or conceivably a regularly updated snapshot thereof) for unauthenticated users and bots.
Bots, scrapers, and scammers are not (I hope) authenticated users, so they'd be stuck with (2) which might be slow. Meanwhile all of us authenticated users would be using the lightly loaded (1) server.
Would that not be a simple solution? But I'm not an infrastructure guy, so I'm probably just blowing hot air.
Simon
On Thu, 6 Aug 2026 at 17:58, Magnus Viernickel via ghc-devs < ghc-devs@haskell.org> wrote:
Dear GHC Devs,
I have some interesting information to share with you. You might have noticed that our gitlab was a bit more swamped the recent days again and today I investigated and discovered that Anubis was getting bypassed by requests with a new user agent that I haven't seen before, "Lightpanda/1.0".
Normally, anubis checks for user agents that are known bot user agents and blocks them; Lightpanda was left unblocked because Anubis only filtered for LightPanda (capital p).
This is normally not a problem because most crawlers try to use user agents that look like browsers; if you're using a browser user agent, you're getting a higher weight and you get presented as challenge; but you're also much harder to sift out because you look benign.
While the amount of requests we were getting with that user agent had reached absurd dimensions, this was also a blessing. First of all, it was easy to ban it and reduce the gitlab load drastically.
Second, now we could run an analysis on who is using these malicious clients. Xe, the creator of Anubis kindly provided me with a tool that qualitatively (based on unique IP addresses, disregarding hits per IP) puts IP addresses in buckets based on several known factors, e.g. location or the ASN (which entity the IP address block is delegated to).
The results can be seen here: https://gitlab.haskell.org/-/snippets/6132
A few noteworthy things: - 1.4 Million distinct IPv4 addresses have been used with this bot user agent - the IP addresses stem from 225 different countries. I think we have ~ 30 more distinct ISO country codes. - we have some known faces! like spacex which got flagged 100% time on their 21000 distinct IP addresses they're using to DDoS us - we also have quite a lot of domestic use ASNs, which indicates that botnets are involved, too! - Xe noted, that we can proudly say this counts as a DDoS
Mind that this does not include all the spammers that are actually trying to hide the fact that they're spammers, which still exist. Also mind that this was just the data from ~1 day of logs.
I hope this gives a nice sense of what we're dealing with and maybe it also helps with your frustration to see my frustration and the fact that we're sometimes getting little wins. Like today.
Best
Magnus
_______________________________________________ ghc-devs mailing list -- ghc-devs@haskell.org To unsubscribe send an email to ghc-devs-leave@haskell.org