How I protect my Forgejo instance from AI web crawlers
17 comments
·December 21, 2025maelito
I'm having lots of connections every day from Singapor. It's now the main country... despite the whole website being French-only. AI crawlers, for sure.
Thanks for this tip.
input_sh
Fun fact: you don't get rid of them even when you put a captcha on all visitors from Singapore. I still a the spike in traffic that perfectly matches the spike in served captchas, but this time it's geographically distributed between places like Iraq, Bangladesh and Brazil.
Hopefully it at least costs them a little bit more.
arjie
Amazonbot does this despite my efforts in robots.txt to help it out. I look at all the Singapore requests and they’re Amazonbot trying to get various variants of the Special:RecentChanges page. You’re wasting your time, Amazonbot. I’m trying to help you.
userbinator
Unfortunately this means, my website could only be seen if you enable javascript in your browser.
Or have a web-proxy that matches on the pattern and extracts the cookie automatically. ;-)
apples_oranges
HTTP 412 would be better I guess..
reconnecting
tirreno (1) guy here.
Our open-source system can block IP addresses based on rules triggered by specific behavior.
Can you elaborate on what exact type of crawlers you would like to block? Like, a leaky bucket of a certain number of requests per minute?
reconnecting
I believe there is a slight misunderstanding regarding the role of 'AI crawlers'.
Bad crawlers have been there since the very beginning. Some of them looking for known vulnerabilities, some scraping content for third-party services. Most of them have spoofed UAs to pretend to be legitimate bots.
Therefore, our current strategy for bad crawler mitigation is first of all to flag HEAD requests, then to check if a bot has multiple 404 errors or requests vulnerable URLs like wp-admin, etc., and finally to check for bad UAs (like Go-Client 1.0).
This is approximately 30–40% of traffic on any website.
notachatbot123
The article is about AI web crawlers. How can your tool help and how would one set it up for this specific context?
reconnecting
I don't see how an AI crawler is different from any others.
The simplest approach is to count the UA as risky or flag multiple 404 errors or HEAD requests, and block on that. Those are rules we already have out of the box.
It's open source, there's no pain in writing specific rules for rate limiting, thus my question.
Plus, we have developed a dashboard for manually choosing UA blocks based on name, but we're still not sure if this is something that would be really helpful for website operators.
csilker
Cloudflare has a solution to protect routes from crawlers.
roywashere
Sure, but the whole point of self-hosting forgejo is to not use these big cloud solutions. Introducing cloudflare is a step back!
immibis
My issue with Gitea (which Forgejo is a fork of) was that crawlers would hit the "download repository as zip" link over and over. Each access creates a new zip file on disk which is never cleaned up. I disabled that (by setting the temporary zip directory to read-only, so the feature won't work) and haven't had a problem since then.
It's easy to assume "I received a lot of requests, therefore the problem is too many requests" but you can successfully handle many requests.
This is a clever way of doing a minimally invasive botwall though - I like it.
userbinator
Each access creates a new zip file on disk which is never cleaned up.
That sounds like a bug.
bob1029
> you can successfully handle many requests.
There is a point where your web server becomes fast enough that the scraping problem becomes irrelevant. Especially at the scale of a self-hosted forge with a constrained audience. I find this to be a much easier path.
I wish we could find a way to not conflate the intellectual property concerns with the technological performance concerns. It seems like this is essential to keeping the AI scraping drama going in many ways. We can definitely make the self hosted git forge so fast that anything short of ~a federal crime would have no meaningful effect.
idontsee
> There is a point where your web server becomes fast enough that the scraping problem becomes irrelevant.
It isn't just the volume of requests, but also bandwidth. There have been cases where scraping represents >80% of a forge's bandwidth usage. I wouldn't want that to happen to the one I host at home.
spockz
Maybe it is fast enough but my objection is mostly due to the gross inefficiency of crawlers. Requesting downloads of whole repositories over and over, leading to storing these archives on disk wasting CPU cycles to create them and storage space to retain them, and bandwidth to sent them over the wire. Add this to the gross power consumption of AI and hogging of physical compute hardware, and it is easy to see “AI” as wasteful.
Can someone help me understand where all this traffic is coming from? Are there thousands of companies all doing it simultaneously? How come even small sites get hammered constantly? At some point haven't you scraped the whole thing?