Using lots of little tools to aggressively reject the bots

1. I fear you may be underestimating the volume of bot traffic websites are now receiving. I recommend reading this article to get an idea of the scale of the problem: https://thelibre.news/foss-infrastructure-is-under-attack-by...

2. Not all requests are created equal. 20 requests a second for the same static HTML file? No problem. But if you have, say, a release page for an open source project with binary download links for all past versions for multiple platforms, each one being a multi megabyte blob, and a scraper starts hitting these links, you will run into bandwidth problems very quickly, unless you live in a utopia where bandwidth is free.

3. You are underestimating the difficulty of caching dynamic pages. Cache invalidation is hard, they say. One notably problematic example is Git blames. So far I am not aware of any existing solution for caching blames, and jury rigging your own will likely not be any easier than the “solution” explored in the TFA.

A friend of mine had over 1000 requests/sec on his Gitea at peaks. Also, you aren't taking into account some of us don't have a "server", just some shitbox computer in the basement.

This isn't about mere dozen requests. It gets pretty bad. It also slows down his life.

I sympathize with the general gist of your post, but I've seen many a bot generate more traffic than legitimate users on our site.

Never had any actual performance issue, but I can see why a site that expects generally a very low traffic rate might freak out. Could they better optimize their sites? Probably, I know ours sucks big time. But in the era of autoscaling workloads on someone else's computer, a misconfigured site could rack up a big ass bill.

It's not fuck the bots, it's fuck the bot owners for using the websites as they want, and not at minimum, asking. Like 'hey cool if I use this tool to interact with your site for this and that reason?'

No, they just do it. So that can scrape data, which at this point in time for AI which has hit the cap on what it can consume knowledge wise, scrapes it because live updates and new information is most valuable to them.

So they will find tricky, evil ways to hammer resources that we as site operators own; even minimally to use site data to their profit, their success, their benefits while blatantly saying 'screw you' as they ignore robots.txt or pretend to be legitimate users.

There's a digital battle field going on. Clients are coming in as real users using IP lists like from https://infatica.io/

A writeup posted to HN about it

https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-...

A system and site operator has every right to build the tools they want to protect their systems, data, and have a user experience that benefits their audiences.

Your points are valid and make sense, but; it's not about that. It's about valuing authentic works, intellectual properties, and some dweeb that wants to steal it doesn't get to just run their bots against resources at others detriments, and their benefits.

He said in the article there were requests that made a tarball of an entire repository, for each sha the git tree. No matter how you slice it that's pretty ugly.

Sure, you could require any number of (user hostile) interactions (logins, captchas, etc) to do expensive operations like that, but now the usability is compromised which sucks.

> 20 requests per second is just... Nothing.

Unless you're running mediawiki.

Are there easy settings I should be messing with for that?

Can you expand on the better ways of handling bots? Genuinely curious.

If your robots.txt has a line specifying, for example

   Disallow: /private/beware.zip

But in my experience it isn't the robots.txt violations being so flagrant (half the requests are probably humans who were curious what you're hiding, and most bots written specifically for LLMs don't even check the robots.txt). The real abuse is the crawler that hits an expensive and frequently-changing URL more often than reasonable, and the card-testers hitting payment endpoints, sometimes with excessive chargebacks. And port-scanners, but those are a minor annoyance if your network setup is decent. And email spoofers who bring your server's reputation down if you don't set things up correctly early on and whenever changing hosts.

Very curious if a bad actor can sue you if you serve them a zip bomb from an EU server. Got any links?

That doesn't logic.

Giving someone a "financial incentive" to do something (by gasp using a search engine, or comparing prices) does not make that thing ethical or cool to do in and of itself.

I wonder where you ever got the idea that it does.

[flagged]

My experience with them has been the same. On one of my employer's websites, the top three bots were Google, Bytedance and Semrush. It is a small website for a niche audience, not even in English, and changes very infrequently (like once or twice a quarter). That did not stop these three bots from hammering the site, every second

From the article it's sure starting to seem like people across the internet are just starting to realize what happens when you don't have just 3-4 search engines responsible for crawling for data anymore. When data becomes truly democratized, its access increases dramatically, and we can either adjust or shelter ourselves while the world moves on without us.

Did Google never ever scrape individual commits from Gitea?

They had those stupid tags telling them far more anyway. Nevermind what it was doing to time to first paint/interactive.

They’re using the sign up and sign in forms, and also the search, and then clicking on those search results. I thought some bad actor is masquerading as AI scrapers to enumerate accounts, but their behavior is consistent with a scraper.

> They also respect robots.txt

All the reports I've heard from organizations dealing with AI crawler bots say they are not honest about their user agent and do not respect robots.txt

"It's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more." https://xeiaso.net/notes/2025/amazon-crawler/

We ended up writing similar rules to the article. It was just based on frequency.

While we were rate limiting bots based on UA, we ended up also having to apply wider rules because traffic started spiking from other places.

I can't say if it's the traffic shifting, but there's definitely a big amount of automated traffic not identifying itself properly.

If you look at all your web properties, look at historic traffic to calculate <hits per IP> in <time period>. Then look at the new data and see how it's shifting. You should be able to identify the real traffic and the automated very quickly.

> While our platform is primarily designed for live, logged-in users, it also works well for bot detection and blocking.

tirreno is designed to work with logged-in users, so all actions are tied to usernames rather than IP addresses. From this perspective, we strongly avoid [1] making any decisions or taking actions based solely on IP address information.

[1] https://www.tirreno.com/bat/?post=2025-05-25

You're right, blanket blocking based on IP ranges (like TOR or DC) often creates false positives and punishes privacy-conscious users. Therefore, unlike the traditional way of blocking an IP just because it is from a data center, tirreno suggests using a risk-based system that takes into account dozens or hundreds of rules.

As in my example, if the IP is from a data center and creates a lot of 404 errors, send it to a manual review queue or to automatic blocking (not recommended).

Personally, I prefer to manually review even bot activity, and tirreno, even if it's not directly designed for bot management, works great for this, especially in cases when bad bots are hidden behind legitimate bot UA's.

Increasing amounts of traffic are flowing through Google VPN if you’re on a Pixel, and on Apple, there is iCloud Private Relay. I’d have thought the address-neighbors issue would be especially likely to catch out these situations?

Overly simplistic solutions like this absolutely will actively cost you real customers and real revenue.

I think it depends on what you mean by authorized users. If you're saying everyone is authorized then relying on public scrapers isn't going to work with unlisted pages. Search engines are bots, too.

I'm thinking more if it's people coming from email or QR code or link from other sites that keep bots out. It can also be more about sequence of accessing pages. The idea is having behavior unlock the site. Because it should be pretty easy to cause people and bots to behave differently.

For example, you could set it up such that access to gitea requires simply accessing the documentation site first.

It can be less unfriendly than requiring them to create an account and login. You could send 403 instead.

Yep! Eye opening report on residential proxies from Trend Micro.

https://www.trendmicro.com/vinfo/us/security/news/vulnerabil...

And? If your network allows malware you should be blocked.

Anubis author here. I have been working on a no-js approach. I just have to be careful so that the scrapers don't just implement it and bypass it forever.