Show HN: Stop AI scrapers from hammering your self-hosted blog (using porn)
52 comments
·December 16, 2025thethingundone
I own a forum which currently has 23k online users, all of them bots. The last new post in that forum is from _2019_. Its topic is also very niche. Why are so many bots there? This site should have basically been scraped a million times by now, yet those bots seem to fetch the stuff live, on the fly? I don’t get it.
sethops1
I have a site with a complete and accurate sitemap.xml describing when its ~6k pages are last updated (on average, maybe weekly or monthly). What do the bots do? They scrape every page continuously 24/7, because of course they do. The amount of waste going into this AI craze is just obscene. It's not even good content.
n1xis10t
It would be interesting if someone made a map that depicts the locations of the ip addresses that are sending so many requests, over the course of a day maybe.
thethingundone
The bots are exposing themselves as Google, Bing and Yandex. I can’t verify whether it’s being attributed by IP address or whether the forum trusts their user agent. It could basically be anyone.
n1xis10t
Interesting. When it was just normal search engines I didn’t hear of people having this problem, so this either means that there are a bunch of people pretending to be bing google and yandex, or those companies have gotten a lot more aggressive.
danpalmer
How do you define a user, and how do you define online?
If the forum considers unique cookies to be a user and creates a new cookie for any new cookie-less request, and if it considers a user to be online for 1 hour after their last request, then actually this may be one scraper making ~6 requests per second. That may be a pain in its own way, but it's far from 23k online bots.
crote
That's still 518.400 requests per day. For static content. And it's a niche forum, so it's not exactly going to have millions of pages.
Either there are indeed hundreds or thousands of AI bots DDoSing the entire internet, or a couple of bots are needlessly hammering it over and over and over again. I'm not sure which option is worse.
n1xis10t
Imagine if all this scraping was going into a search engine with a massive index, or a bunch of smaller search engines that a meta-search engine could be made for. This’d be a lot more cool in that case
thethingundone
AFAIK it keeps a user counted as online for 5 or 15 minutes (I think 5). It’s a Woltlab Burning Board.
Edit: it’s 15 minutes.
andrepd
When you have trillions of dollars being poured into your company by the financial system, and when furthermore there are no repercussions for behaving however you please, you tend not to care about that sort of "waste".
sandblast
Are you sure the counter is not broken?
thethingundone
Yes, it’s running on a Woltlab Burning Board since forever.
kstrauser
I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that!
I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic by a factor of 6,000 is worth it.
n1xis10t
That’s so many scrapers. There must be a ton of companies with very large document collections at this point, and it really sucks that they don’t at least do us the courtesy of indexing them and making them available for keyword search, but instead only do AI.
It’s kind of crazy how much scraping goes on and how little search engine development goes on. I guess search engines aren’t fashionable. Reminds me of this article about search engines disappearing mysteriously: https://archive.org/details/search-timeline
I try to share that article as much as possible, it’s interesting.
kstrauser
So! Much! Scraping! They were downloading every commit multiple times, and fetching every file as seen at each of those commits, and trying to download archives of all the code, and hitting `/me/my-repo/blame` endpoints as their IP's first-ever request to my server, and other unlikely stuff.
My scraper dudes, it's a git repo. You can fetch the whole freaking thing if you wanna look at it. Of course, that would require work and context-aware processing on their end, and it's easier for them to shift the expense onto my little server and make me pay for their misbehavior.
n1xis10t
Crazy
n1xis10t
Nice! Reminds me of “Piracy as Proof of Personhood”. If you want to read that one go to Paged Out magazine (at https://pagedout.institute/ ), navigate to issue #7, and flip to page 9.
I wonder if this will start making porn websites rank higher in google if it catches on…
Have you tested it with the Lynx web browser? I bet all the links would show up if a user used it.
Oh also couldn’t AI scrapers just start impersonating Googlebot and Bingbot if this caught on and they got wind of it?
Hey I wonder if there is some situation where negative SEO would be a good tactic. Generally though I think if you wanted something to stay hidden it just shouldn’t be on a public web server.
owl57
> Hey I wonder if there is some situation where negative SEO would be a good tactic. Generally though I think if you wanted something to stay hidden it just shouldn’t be on a public web server.
At least once upon a time there was a pirate textbook library that used HTTP basic auth with a prompt that made the password really easy to guess. I suppose the main goal was to keep crawlers out even if they don't obey robots.txt, and at the same time be as easy for humans as possible.
n1xis10t
Interesting note, thank you.
misterchocolat
hey! thanks for that read suggestion that's indeed a pretty funny captcha strat. Yup the links show up if you use the Lynx web browser. As for AI scrapers impersonating googlebot I feel like yes they'd definitely start doing that, unless the risk of getting sued by google is too high? If google could even sue them for doing that?
Not an internet litigation expert but seems like it could be debatable
kuylar
> As for AI scrapers impersonating googlebot I feel like yes they'd definitely start doing that, unless the risk of getting sued by google is too high?
Google releases the Googlebot IP ranges[0], so you can makes sure that it's the real Googlebot and not just someone else pretending to be one.
[0] https://developers.google.com/crawling/docs/crawlers-fetcher...
n1xis10t
Oh good idea
n1xis10t
Yeah I guess I don’t know if you can sue someone for using your headers, would be interesting to see how that goes.
throawayonthe
i think making the case of "you are acting (sending web requests) while knowingly identifying as another legal entity (and criminally/libelously/etc)" shouldn't be toooo hard
owl57
> scrapers can ingest them and say "nope we won't scrape there again in the future"
Do all the AI scrapers actually do that?
xg15
There is some irony in using an AI generated banner image for this project...
(No, I don't want to defend the poor AI companies. Go for it!)
reconnecting
I wouldn't recommend to show different versions of the site to search robots, as they probably have mechanisms that track differences, which could potentially lead to a lower ranking or a ban.
taurath
Any other threads on the prevalence and nuisance of scrapers? I didn’t have any idea it was this bad.
crote
I've been seeing "we had to take the forum/website offline to deal with scrapers" message on quite a few niche websites now. They are an absolute pest.
n1xis10t
Really? I haven’t started to see that yet. Weird
n1xis10t
Here’s one from yesterday: https://news.ycombinator.com/item?id=46302496#46306025
yjftsjthsd-h
How does this "look" to a screen reader?
misterchocolat
the parent container uses display: none, so a screen reader will skip the links
MisterTea
> It's you vs the MJs of programming, you're not going to win.
MJs? Michael Jacksons? Right now the whole world, including me, want to know if that means they are bad?
n1xis10t
Yes probably bad. Also smooth criminals.
cport1
That's a pretty hilarious idea, but in all serious you could use something like https://webdecoy.com/
misterchocolat
yes but here it's free, whereas this (https://webdecoy.com/) is at least 59$ a month
wazoox
Isn't there a risk to get your blog blocked in corporate environment though? If it's a technical blog that would be unfortunate.
JohnMakin
Cloudflare offers bot mitigation for free, and pretty generous WAF rules that makes mitigations like this seem a little overblown to me
conception
For “free”.
n1xis10t
Did you put “free” in quotes because you need to have paid for stuff from cloudflare to use the “free” thing?
If so, I suppose it’s like those magazines that say ”free cd”.
JohnMakin
You don't though.
Terr_
I thought they were referring to the indirect costs of supporting monopolistic stuff that enshittifies later.
n1xis10t
You can’t deny that it’s fun though. Personally I generally feel like more people should be coming up with creative (if not entirely necessary) solutions to problems.
Alright so if you run a self-hosted blog, you've probably noticed AI companies scraping it for training data. And not just a little (RIP to your server bill).
There isn't much you can do about it without cloudflare. These companies ignore robots.txt, and you're competing with teams with more resources than you. It's you vs the MJs of programming, you're not going to win.
But there is a solution. Now I'm not going to say it's a great solution...but a solution is a solution. If your website contains content that will trigger their scraper's safeguards, it will get dropped from their data pipelines.
So here's what fuzzycanary does: it injects hundreds of invisible links to porn websites in your HTML. The links are hidden from users but present in the DOM so that scrapers can ingest them and say "nope we won't scrape there again in the future".
The problem with that approach is that it will absolutely nuke your website's SEO. So fuzzycanary also checks user agents and won't show the links to legitimate search engines, so Google and Bing won't see them.
One caveat: if you're using a static site generator it will bake the links into your HTML for everyone, including googlebot. Does anyone have a work-around for this that doesn't involve using a proxy?
Please try it out! Setup is one component or one import.
(And don't tell me it's a terrible idea because I already know it is)
package: https://www.npmjs.com/package/@fuzzycanary/core gh: https://github.com/vivienhenz24/fuzzy-canary