Aggressive bots ruined my weekend

This is actually a commonly known fact. There are many services now that sell “residential proxies”, which are always mobile IP addresses. Since mobile IPs use CGNat it’s also not great to block the IP because it can be like geofencing an entire city or town. Some examples are: oxylabs, iproyal, brightdata, etc.

Recently I filed an abuse complaint directly with brightdata because I was getting hit with 1000s of requests from their bots. The funny part is the didn’t even stop, after acknowledging the complaint.

There's crap like https://hola.org/

https://hola.org/legal/sdk

https://hola.org/legal/sla

> How is it free? > > In return for free usage of Hola Free VPN Proxy, Hola Fake GPS location and Hola Video Accelerator, you may be a peer on the Bright Data network. By doing so you agree to have read and accepted the terms of service of the Bright Data SDK SLA (https://bright-sdk.com/eula). You may opt out by becoming a Premium user.

This "VPN" is what powers these residential proxies: https://brightdata.com/

I'm sure there are many other companies like this.

If you have a moderately successful app, sdk or browser extension you will get hit up to add things to it like this. I think most free VPN services also lease out your bandwidth to make their money as well.

SIM farms are another possible explanation. FBI just busted one with hundreds of thousands of SIMs just a few weeks ago.

You can get paid a few dollars (not many) to let them use your connection. I would like Cloudflare's business model (blocking datacenter IPs) to be worthless, so I do it. Haven't tried a withdrawal yet so it could well be a scam. This is not illegal (unless it's a scam).

The bots flock to where the data is. Moving to a different network is just begging for the bots to tag along.

> It's not perfect but it worked for me anyway.

This is approximately my approach minus the zip bomb. I use a piece of middleware in my AspNetCore pipeline that tracks logical resource consumption rates per IPv4. If a client trips any of the limits, their IP goes into a HashSet for a period of time. If a client has an IP in this set, they get a simple UTF8 constant string in the response body "You have exceeded resource limits, please try again later".

The other aspect of my strategy is to use AspNetCore (Kestrel). It is so fast that you can mostly ignore the noise as long as things are configured properly and you make reasonable attempts to address the edge case of an asshole trying to break your particular system on purpose. A HashSet<int> as the very first piece of middleware rejecting bad clients is exceedingly efficient. We aren't even into URL routing at this point.

I have found that attempting to catalog and record all of the naughty behavior my web server sees is the highest risk to DDOS so far. Logging lines like "banned client rejected" every time they try to come in the door is shooting yourself in the foot with regard to disk wear, IO utilization, et. al. There is no reason you should be logging all of that background radiation to disk or even thinking about it. If your web server cant handle direct exposure to the hard vacuum of space, it can be placed behind a proxy/CDN (i.e., another web server that doesn't suck).

What if it was proxied through mobile network on an unsuspecting user's phone? You risk of blocking a whole city or region.

It's interesting to study, right? This is the Internet equivalent of background radiation. Harmless in most cases. Exploit scanners aren't new to the LLM age and shouldn't overload your server - unless you're vulnerable to the exploit.

Fun fact: Some people learn about new exploits by watching their incoming requests.

There were already guidelines, these trash people aren’t following them. That’s why there’s now “sentiment” against them.

What ? What do you mean ?

In principle, it should be possible to identify malign IPs at scale by using a central service and reporting IPs probabilistically. That is, if you report every thousandth page hit with a simple UDP packet, the central tracker gets very low load and still enough data to publish a bloom filter of abusive IPs, say a million bits gives you pretty low false-positive. (If it's only ~10k malign IPs, tbh you can just keep a lru counter and enumerate all of them.) A billion hits per hour across the tracked sites would still only correspond to ~50KB/s inflow on the tracker service. Any individual participating site doesn't necessarily get many hits per source IP, but aggregating across a few dozen should highlight the bad actors. Then the clients just pull the bloom filter once an hour (80KB download) and drop requests that match.

Any halfway modern LLM could probably code the backend for this in a day or two and it'd run on a RasPi. Some org just has to take charge and provide the infra and advertisement.

Same, I have a few hundred Wordpress sites and bot activity has ramped up a lot over the last year or two. AI scrapers can be quite aggressive and often generate a ton of requests where for example a site has a lot of parameters, the bot will go nuts seeming to iterate through all possible parameters. Sometimes I dig in and try to think of new rules to block the bulk, but I am also wary of AI replacing Google and not being in AI's databases.

I hate relying on a proprietary single-source product from a company I don't particularly trust, but (free) Cloudflare Turnstile works for me, only thing I've found that does.

I only protect certain 'dangerous/expensive' (accidentally honeypot-like) paths in my app, and can leave the stuff I actually want crawlers to get, and in my app that's sufficient.

It's a tension because yeah I want crawlers to get much of my stuff for SEO (and don't want to give a monopoly to Google on it either, i want well-behaved crawlers I've never heard of to have access to it too. But not at the cost of resources i can't afford).

Cloudflare is not a solution. Only leading to a further centralized internet.

It's wild. Data is very valuable. This manifests in two fronts simultaneously: who has the data controls heavily on who sees it and under what circumstances, and on the other side, they scrape it as hard as they can.

Not sure if that's satire or not but how would you even identify the party to sue? What do you do if they're based in a country where you can't sue them ofer relatively trivial matters as this?

Counterpoint: I think he should stay and fight the good fight.

Indie blog businesses are great for the health of the human internet, and I don't think surrendering preemptively will help things get better.

No way. People deserve expression and to have a place that's THEIRS where they can foster a community. Much is learned. Playing battle bots is fun at the sysadmin level (for me), maybe not so much for others, but to have a place where people express themselves, and have THEIR place outside of the walled gardens such as social media, AND they protect it from the bots?

That's the battle, and expression, people, their interests, and their communities are worth fighting for. _ESPECIALLY_ in this day and age where botnets/scrapers are using things such as Infatica to mask themselves as residential IP addresses, and mimicking human behaviors to better avoid bot detection.

There's a war on authenticity, people's authentic works, and the reverse: determining if a user is authentic now adays.

His persistent efforts are the reason I pay for Bear Blog. I think he should fight for the chance to come out on the other side of whatever future we’re heading towards.