Google is using AI to censor independent websites like mine

The web isn't what it was 10 or 15 years ago. Back then, you could stumble across countless useful and unique websites on just about any topic—travel included.

Today, it's different. I do think Google played a major role in shrinking the visibility of the independent web. If Google only shows 300 links per search (30 pages x 10 results), anything beyond that effectively doesn't exist for most users.

Some might argue that only the top few results matter, but I disagree. For broad topics like "video games," "war," or "love," you'd expect there to be thousands of worthwhile links—not just the same recycled content from YouTube, TikTok, Reddit, or SEO-heavy content farms.

As a side project, I run a hobby web crawler to explore what's still out there. That's where my perspective comes from regarding the current state of the open web:

https://github.com/rumca-js/Internet-Places-Database

It doesn't hinge entirely on that. There's a lot of ambient background context here too.

The idea Google is hostile to long tail indie content isn't exactly a groundbreaking claim, it's been obvious and widely discussed for years. They've been losing the original culture for a long time. Google circa 2000-2010 was very libertarian. It believed in a large decentralized web in which Google helped all users with all queries, without passing judgement. If it was obscure and you wanted it, Google would reliably surface it in the first page of results every single time. This was the Google that believed in the indie web so much it purchased Blogger.

Starting around 2010-2012 the rate at which they hired new grads went up quite sharply (I was there and saw it). The average level of experience dropped sharply. These recruits brought with them the new authoritarian politics of the university campus. Around 2015-2016 you start to see Google start to just openly engage in political activism, tossing the hard-won reputation for neutrality in the trash. Unfortunately, this new worldview was incompatible with the prior commitment to the indie web. Whereas the Google of Matt Cutts cared a lot about surfacing tiny sites, the new Google became highly suspicious of any content that wasn't from sources they deemed "reliable", "authoritative" etc [1]. They defined these terms to mean basically any large left-leaning source, without reference to objective metrics. Put simply: if it's on .gov, .edu or one hop removed then it's reliable, if it's not then it isn't.

This shows up in how easy it now is to find queries where Google gives you the exact opposite of what you're asking for, no matter how clearly you specify the search terms. This would have once been considered a high severity code yellow, now it's by design. The open web won out over AOL partly because old Google fostered it, but one gets the feeling that Google now views its child with disgust. Can you imagine Google purchasing Substack, as they once did with Blogger? It's unthinkable. They'd undoubtably view it as a hive of villainy and scum. In the event they did buy it the first thing they'd do is delete most of its content.

Unfortunately, you can't be both anti-misinformation and pro-open-web. These two things are irreconcilable. Either the world is complex and anyone might have insight to contribute, or it's simple and the right answer is always found via traversing a shallow hierarchy of trusted sources.

So: does your random indie travel blog "demonstrate expertise" or "authoritativeness" as defined by someone who has been through the Ivy League universities? No. Are these the sorts of sites that can eventually become big and a recognized source of authoritative expertise, given enough nourishment from the watering can of unbiased search? Yes! That's how the web grew to start with. But Google doesn't care anymore and with the loss of its primary patron the open web is in its twilight years. As the author says: he was invited to Google HQ to hear an apology, and also to be told nothing will change. The new web is no different to AOL except in minor technical details, because that's how the woke generation like it.

[1] e.g. https://support.google.com/websearch/answer/12395529?hl=en

> My hunch is Google never psychologically recovered from Facebook absolutely wiping out G+, and they have been on a mission to ensure nothing like that ever happens again.

Same for Microsoft with phones.

That's why Google and Microsoft are trying to use AI to crush us while Apple is like "I'm pretty content with how much money I make".

the funniest part here is that if G+ is part of the reason, they still don't understand why people didn't want to use it, making any decisions which follow wrong &/or misguided. while I'm not terribly in favor of breaking up companies, there is far too much incentive for Google, given its ad business (which is really what the company now is), to have any voice in the future of the Web. Chrome must be spun off, & firewalls put in place between the ad side & the search side of things (unless, of course, they really are just a govt subsidiary & this has been the plan all along, but there would have to be tangible proof of that).

>My hunch is Google never psychologically recovered from Facebook absolutely wiping out G+, and they have been on a mission to ensure nothing like that ever happens again.

That's an orthogonal issue to me. IMHO, modern Google never really cared about social media platforms because it never really understood them. Otherwise it would have bought Facebook, Instagram, WhatsApp, Snapchat, Tiktok, Twitter, early on before they became industry titans, the same way they bought Youtube, Maps, etc in the early days.

They've been coasting for so long on the search ad revenue money printer, that they're blind to everything else going on around them, so they're always reacting instead of proactive, but always too late.

> This uncited, anonymous quote sounds very made up. Cursory search couldn't find anything like this.

Quote is from Noam Shazeer (of transformer fame) in the Dwarkesh Podcast: https://www.dwarkesh.com/p/jeff-dean-and-noam-shazeer

Two minutes in a search engine that is Not Google:

> Shazeer says he’s excited about Google expanding its focus to include helping users create new AI-generated content. “Organizing information is clearly a trillion-dollar opportunity, but a trillion dollars is not cool anymore,” he said recently on a podcast[2]. “What's cool is a quadrillion dollars.”

[1] https://www.wired.com/story/google-openai-gemini-chatgpt-art...

[2] at minute 29:18 , https://www.dwarkesh.com/p/jeff-dean-and-noam-shazeer

Noam Shazeer says it at some point in the Dwarkesh Patel podcast.

Edit : Saw multiple people comment by the time I hit sent. Was it also memorable for you guys? I think it was just the way he said it or something.

https://www.youtube.com/watch?v=v0gjI__RyCY

Cursory google search quadrillion dollars "Organizing information" brings up multiple sources on the first page

> Google gave you incredible monetizable traffic for two+ decades.

You forgot to add "for free" /s

(I mean, really? Google made billions on being a better browser than Yahoo!, AltaVista, AOL Search and whatever was there. They build up people's trust only to abuse it to the limits in practically every area they could get away with.)

[dead]

It’s appallingly bad faith.

Google have a page where they give advice on “Creating helpful, reliable, people-first content”:

https://developers.google.com/search/docs/fundamentals/creat...

Things like “Provide a great page experience”, “Focus on people-first content”, and “Avoid creating search engine-first content”.

This article reframes it as an unfulfilled obligation and betrayal because ‘Google Promised to Reward Publishers Who Invest Resources Into Content Created “By People, For People”’

> Google tunes its algorithms and lets the chips fall

This is even more sloppy and could be easily applied to any real censorship, where some (censored) lose and some (not censored) win

If you're "tuning" your algorithm with the goal of blocking all small independent websites, then yeah, you're doing the shadow (if you don't tell the sites about it) banning. By ignoring the intent of the policy and replacing it with a truism you don't really have a terminology counter argument

It's a literal shadow ban and Google writes the new algorithms, talks about it internally and decides what type of sites it wants to down rank and replace with AI Answers.

The algorithm isn't some mystery formula it is literally what Google decides. There is no someone wins someone loses. Google wants to provide the answers and is taking market share from smaller publisher today and larger publishers tomorrow.

He says that when he was invited to Google to discuss the matter, they said:

And Google gave us a clear and unequivocal apology. Google said our sites didn’t deserve our shadowbans, and that it wasn’t our fault.

Whether Google itself used the term "shadowban" or not, they clearly acknowledged the drastic effect of their algorithms that favor "AI". So we know what he means.

if using algorithm to promote some information and suppress some other isn't censorship to you, then honestly your definition of censorship is narrow to the point of being useless.

>Honest people making honest statements don't need to exaggerate to make the point.

First i want to congratulate you on waking up from your 10 year coma.

That statement is no longe true. Nowadays you have to lie and exaggerate to make a point. That's just how things work

[flagged]

And I'm tired of this sort of sloppy and inflammatory discrediting.

It fits 'shadowbanning' to an absolute tee. Without any kind of notification, or any recourse, a switch is flicked to "off" in a single moment, putting a domain on the shitlist. This is now very well documented, including by this post, and incredibly obvious from the graph.

> Our creators actually visit the places we write about.

> Our creators take and publish thousands of original photos.

Not defending the site, but I guess tons of human created content will be lost? Have fun with LLM hallucinated travel guides and AI generated travel photos.

At least we'll soon get tour guides for our next moon vacation!

Stealing other people's output and then cutting them off from discovery isn't a right either.

Since politicians do not protect us from these criminals, we are fortunate that no one except for CEOs under the influence of cocaine wants "AI".

And where the training data for your AI advice comes from?

> The article treats as axiom that organic traffic is a right. It's not.

Agreed, but traffic has been the substitute for monetary rewards so far. If they aren't sending the traffic then someone is going to start to need to pay for surfacing and verifying information directly.

I haven't been able to get much practical travel advice from AI yet, any tips?

For stuff like hiking routes, permitting, local cuisine, etc. I don't think it can replace a good blog from an experienced person at this point.

What's your reference re. what most people want? Or at least a reference that AI advice is better, more specific, and its references aren't fake?

I think the days of Google driving traffic to sites then splitting the advertising revenue (or missing out entirely) are largely over. From Google's point of view, they are just closing the loop.

It is a shame because the mutually beneficial relationship between Google and sites has driven a huge amount of the tech economy. Sometimes this has been bad (and advertising is a scourge for many reasons, I can't believe I am defending it) but in general the web would be a lot less useful if people couldn't make money of advertising.

I think long-term sites will stop relying on advertising and go with payments or memberships for information. This will hurt everyone but be disastrous for Google if they cannot spider up-to-date information. It will also hurt the free flow of information that we now enjoy.

What is really galling is that with all the impressive AI summaries, the search results themselves seem to be getting a lot worse[0]. Many times I have used Google and the AI summary is pretty good but the actual page the summary is from is buried well down in the search results. Is this something they are doing deliberately to make the AI summary seem more useful?

[0] An example I blogged about: https://sheep.horse/2025/4/yo_google%2C_thanks_for_the_ai_ov...

> Most people don't want to read an article on travel advice when AI gives us much better and specific advice, with references, when we want it.

Yes, because as we all know Google search's AI has been nothing but a hit. Now if you'll excuse me, I have to go eat my designated 3 small rocks for today.

> don't want to read an article on travel advice when AI gives us

A bit of a tangential anecdote, on the subject of "human written" stuff pre AI.

Circa 99-2000 I knew a US company that would hire students from CEE to write travel content for their many .com properties. The students were instructed to write "as if they'd travelled there", or "as if you're living there", or "in an official tour-guide style", and so on. Judging by how spread the content of my friend's writing was, and how "stereotypical" the other pieces were on those properties (that we found after searching for known content), it was pretty clear that nothing on those sites was genuine. Everything was fake! Since then I take everything with a bit of cynicism, knowing how much of the "natural, organic content" is in fact faked with low-paid contributors. But hey, during those times the gig was pretty good for my friend doing it.

I don’t get this take at all.

The current administration uses bad faith claims of censorship to bully companies into advantaging speech they like (anti-science, anti-trans, anti-democracy, etc). They haven’t shown any interest in actual freedom of speech.

And OP’s use of “censorship” is pretty disingenuous; there is no claim that any free speech topic is in the table. No viewpoint is suppressed because no viewpoint is asserted.

It’s simple commercial lead gen. Google might be engaging in anticompetitive behavior, but this administration is very much a “might makes right” operation, and it’s hard to imagine them siding with a small player with no conservative bona fides and presumably no money to pay to play.

The only way I could imagine OP getting any relief would be to go full MAGA, list a bunch of travel options for hunting gays or whatever, donate at least a million dollars to Trump’s personal coffers, and then tell and scream about oppression of conservatives.

A simple “Walmart thanks suppliers for the market information but will only sell store brand products” case seems incredibly unlikely to get any traction at all.

Basically if you serve information or content. AI or even just some smart coding google can do that too. If you do something with that information Google has not been doing so good at that.

Microsoft in the early 2000s did that very well. They would let you have the data but would gobble up any company that could transform data and make it their own.

But data without applications is useless. Applications without data is also useless.

The applications let us make decisions with our data. Now can AI replace that? Probably in many cases. If it ican then google can just spit out the answer you want.

However, by doing that google may be eating its own lunch. As that ad empire depends on thousands of websites serving up their ad's. If those sites do not exist then what are the ads worth? It was this serving of information/content that drew everyone in. With that scrape of getting ad revenue. Google now can scrape your content and show it above the fold. What reason do you have to make a content farm? But then where does google get the data? They are killing the chicken and the egg at the same time.

> I’m not saying we all have to innovate or perish but how did our rules based order allow Google to get to this point.

Well millions of people learned how to game the Google Search algorithm and created long-winded, hollow content that would rank on the first page to the point that people no longer trusted Google's results.

Then the perfect technology to solve that exact problem came along—one that let Google cease its dependency on the pesky people it was sending traffic to.