Websites and web developers mostly don't care about client-side problems

> The “client-side problems” Siebenmann is talking about are the various anti-bot measures (CAPTCHAs, rate limiters, etc.)

Directly from the article:

> it's not new, and it goes well beyond anti-crawler and anti-robot defenses. As covered by people like Alex Russell, it's routine for websites to ignore most real world client side concerns (also, and including on desktops). Just recently (as of August 2025), Github put out a major update that many people are finding immensely slow even on developer desktops.

The things he links to are about things that are unrelated to anti-bot measures.

The fact is, the web is an increasingly unpleasant place to visit. Users are subject to terrible UX – dark patterns, tracking, consent popups, ads everywhere, etc.

Then along come chatbots and when somebody asks about something, they are given the response on the spot without having to battle their way through all that crap to get what they want.

Of course users are going to flock to chatbots. If a site owner is worried they are losing traffic to chatbots, perhaps they should take a long, hard look at what kind of user experience they are serving up to people.

This is like streaming media all over again. Would you rather buy a legit DVD and wait for it to arrive in the post, then wait through an unskippable lecture about piracy, then wait through unstoppable trailers, then find your way through a weird, horrible DVD menu… or would you rather download it and avoid all that? The thing that alleviated piracy was not locking things down even more, it was making the legitimate route more convenient.

We need to make websites pleasant experiences again, and we can’t do that when we care about everything else more than the user experience.

The Internet and Web were both designed with the assumption of cooperation. I wonder what they would have built if they'd taken hostility into account from day one.

As we've seen security is really hard to build in after the fact. It has to be part of your design concept from the very first, and pervades every other decision you make. If you try to layer security on top you will lose.

Of course you may discover that a genuinely secure system is also unusably inconvenient and you lose to someone willing to take risks, and it's all moot.

Another implicit social contract is that you can tell whether a request is coming from a commercial or non-commercial source based on the originating ISP. This was always a heuristic but it was more reliable in the past.

If 1000 AWS boxes start hammering your API you might raise an eyebrow, but 1000 requests coming from residential ISPs around the world could be an organic surge in demand for your service.

Residential proxy services break this - which has been happening on some level for a long time, but the AI-training-set arms race has driven up demand and thus also supply.

It's quite easy to block all of AWS, for example, but it's less easy to figure out which residential IPs are part of a commercially-operated botnet.

> I think that statement is way too strong and obviously not true of businesses

Amazon.com Inc is currently worth 2.4 billion dollars and the only reason is that most businesses insist on giving their customers the worst online experience possible. I wish that I could one day understand the logic, which goes like this:

1. Notice that people are on their phones all the time.

2. And notice that when people are looking to buy something they first go on the computer or on the smart phone.

3. Therefore let's make the most godawful experience on our website possible, to make sure that our potential customers hate us and don't make a purchase.

4. Customers make their purchase on Amazon instead.

5. Profit??

My forum traffic went up 10x due to bots a few months ago. Never seen anything like it.

> Loading static pages from CDN to scrape training data takes such minimal amounts of resources that it's never going to be a significant part of my costs. Are there cases where this isn't true?

Why did you bring up static pages served by a CDN, the absolute best case scenario, as your reference for how crawler spam might affect server performance?

The following is the best I could collect quickly to provide backup to the statement. Unfortunally it's not the high quality first instance of raw statistics I would have liked.

But from what I have read time to time the crawler acted magnitudes outside of what could have been accepted as just badly configured.

https://herman.bearblog.dev/the-great-scrape/

https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...

https://lwn.net/Articles/1008897/

https://tecnobits.com/en/AI-crawlers-on-Wikipedia-platform-d...

https://boston.conman.org/2025/08/21.1

My estimation is at least 70% of traffic on small sites 300-3000 daily views, is not human

Yes, it’s true. Most sites don’t have a forever cache TTL so a crawler that hits every page on a database-backed site is going to hit mostly uncached pages (and therefore the DB).

I also have a faceted search that some stupid crawler has spent the last month iterating through. Also mostly uncached URLs.

I just had to purchase a cloudflare account to protect two of my sites used for CI that run Jenkins and Gerrit servers. These are resource-hungry java VMs which I have running on a minimally powered server as they are intended to be accessed only by a few people, yet crawlers located in eastern Europe and Asia eventually found it and would regularly drive my CPU up to 500% and make the server unavailable (it should go without saying I have always had a robots.txt on these sites that prohibit all crawling. Such files are a quaint relic of a simpler time). For a couple of years I'd block the various offending IPs, but this past month the crawling resumed again this time intentionally swarmed across hundreds of IP numbers so that I could not easily block them. Cloudflare was able to show me within minutes the entirety of the IP numbers came from a single ASN owned by a very large and well known Chinese company and I blocked the entire ASN. While I could figure out these ASNs manually and get blocklists to add to apache config, Cloudflare makes it super easy showing you the whole thing happening in realtime. You can even tailor the 403 response to send them a custom message, in my case, "ALL of the data you are crawling is on github! Get off these servers and go get it there!" (again sure I could write out httpd config for all of that but who wants to bother). They are definitely providing a really critical service.

It's very real. It's crashed my site a number of times.

[dead]

Heh. Who asked those website owners to have laggy scrolling, non existent contrast, hijack my back button, generally run so much javascript that a cluster is needed client side just to display a 3 line LLM generated blog post?