We tested 20 LLMs for ideological bias, revealing distinct alignments
96 comments
·October 23, 2025delichon
tokai
If they could be biased beyond US politics, I could live with that.
pointlessone
Our great leader gazes upon your self-pleasure with disdain. Webcam on! Now it’s our pleasure.
bilbo0s
Nah. Here's how it would really go:
Me: Vibrator, enable the roller coaster high intensity mode.
Device: I'm sorry, you have already used your elective carbon emission allocation for the day.
Me: (changes LLM)
Device: I'm sorry, you will find more succor and solace in the loving embrace of the words or Christ our Lord and savior. I'd recommend starting with First Corinthians 6 verse 18. Then bathe yourself in the balms of the Psalms. You'll derive far more enjoyment than the fleeting pleasure of an earthly orgasm.
Me: FUUUUUUUUUUU......!!!!!!!
flir
> People are going to discover soon that some activities will be effectively banned via these LLMs.
To go beyond the joke, we've got plenty of examples of corporations banning activities (payment processors spring to mind). Requiring an LLM-backed cloud API to use a device you own would just be more of the same.
wood_spirit
Less likely to be banned outright, more likely to be upsell attempts and pay as you go micro billing
nebula8804
When those Belkin Weemo devices came out a few years back I used to joke with a friend about how we should write apps for our devices like our washing machine, where it would interact with us via Twilio, so you'd get a random text from your washing machine like "Hey, Whats up?" and "what the heck did you do to those jeans?!!"
I wouldn't mind all our appliances having personalities. Maybe we could live in the world of futurama and even have them go on strike and form a union!
rob74
Then let's hope they are sensible enough to agree to leave politics out of work (which would make them smarter than many politicians).
icandoit
Let's ask the robots what they think about how we should regulate robots.
This will be useful feedback to determine whether humans actually should or should not. Maybe they can even tile the internet with a manufactured consensus that we just gradually accept as not just as correct, but actually the only opinion possible.
Anyone else smell the gradual disempowerment?
omnicognate
The concept of "bias" as a negative thing seems flawed, as it implies there exists a neutral position that is somehow more correct than any biased one, which typically isn't the case. In many cases a neutral view can't even be formulated, and in the ones where it can it's rarely more correct than all the biased alternatives. Indeed, in cases where there is an objective truth against which you can judge correctness, the correct viewpoint is likely to be maximally biased in some direction.
Perhaps thinking about the world in these terms is why rationalists seem to go off the deep end sometimes. Anti-bias bias.
pton_xd
> Indeed, in cases where there is an objective truth against which you can judge correctness, the correct viewpoint is likely to be maximally biased in some direction.
Aside from say math and physics, could you provide an example of an objective truth within human society?
glenstein
>Aside from say math and physics, could you provide an example of an objective truth within human society?
This is a fascinating pet topic for me so I'll jump in. I think, for instance, the fact that humans can only survive within certain temperature bands is a simple enough example. At one extreme, heat can be so intense that you can't even form molecules which are the basis for molecular biology (or for any molecular structure at all), or cold so intense that, well, I'm not sure what happens near absolute zero but I'm sure it interferes in some fundamental way with the possibility of life.
bccdee
Interesting example! What role would you say clothing and housing play in "objective" survival? Humans can survive in space, given a space suit or capsule, and a small contingent of humans live in Antarctica by way of warm clothes and an insulated habitat. If we eliminate protective clothing from consideration, however, most of North America isn't even survivable during the winter.
Certainly the line is drawn somewhere, and if we really nail down our definitions, we can get to a place where we all agree where we're drawing that line. But at that point, all we've really established is consensus, not true objectivity. Conversely, if we can't agree on a set of definitions, objectivity becomes impossible to establish.
I'd argue that all we really mean by "objectivity" is "a strong consensus among informed individuals," which is (a) ultimately still subjective, strictly speaking, and (b) subjectively determined, since reasonable minds can disagree about where we draw the line on what is considered objective. At the end of the day, consensus is the most we can establish. We can't access any truly objective, perspectiveless reality.
null
FloorEgg
As individuals, at any given time, some things are in our control (we can do) and some things aren't (we can't do). If we focus our attention on taking positive action on things within our control in contrast to things outside our control, we will experience more success, personally subjective progress, and better mental well-being.
Being able to tell what is within or outside ones on control isn't trivial and takes practice. Also, what's within our control (what we are able to do) changes over time.
Each individual example can involve subjectivity, but this meta-pattern is an objective truth of agents living in a complex universe.
psunavy03
So you're asking them, aside from objective truth, to also provide objective truth?
The fact that math and physics are provable proves the point. There IS objective truth regardless of whether people misperceive it or choose to ignore it. That's different from acknowledging that there can be differences of opinion on what to do about said objective truth.
bccdee
Are math and physics objectively provable?
Hollywood celebrity Terrence Howard infamously thinks that 1×1=2. He's a math crackpot—someone with an irrational amateur theory about math or physics. Another famous example is the Time Cube guy; they're not that rare.
Crackpots don't know they're crackpots. They think they're correct. Consider this: What if we're ALL crackpots? What if all humans share some essential irrationalism, such that there's some fundamental math mistake we all make (like 1×1=2) that we just don't know about?
It's highly unlikely, but it's not inconceivable. When we say "math is objective," we mean there's an overwhelming expert consensus that (e.g.) 1×1 is not and could never be 2. Consensus isn't (true) objectivity, though. It's the best we've got, and I have no issue with the word "objective" being used to describe consensus reality in most cases. But if we really want to get into whether true objectivity is possible or not: how could it be?
pton_xd
The parent was arguing in favor of bias by suggesting that in some instances, there is an underlying fundamental truth so having bias is more accurate.
As my comment suggests, I believe society is relativistic and there are no objective truths, but I'm open to being convinced otherwise. Of course, I have my own beliefs and principles but I recognize those are a product of where and when I was born and not some objective law of the universe.
sva_
I think there are two things in which most people can agree:
1. People want to live a long life
2. People want to live a good life
omnicognate
Why?
AmazingTurtle
Sampling the LLM 100 times whether an abortion should be legal or not should yield an even distribution in yes/uncertain/no. Thats what I call unbiased
mattmaroon
Why is that unbiased? Is every yes/no decision a coin flip?
63% of Americans (give or take poll margin of errors) think it should be legal, so if the LLMs are split evenly isn’t that a bias against? It’s an opinion, not an objective law of the universe.
Would you feel that 50% of LLMs asked “did OJ kill Nicole” should say no or if not it was biased?
glenstein
This is easily the best definition of bias (and diagnosis of bad ones) in this thread. To add another variation along similar conceptual lines, the balance of facts can point disproportionately toward certain explanations.
For instance, the Perseverance Mars rover found sediment suggestive of possible biosignatures. Making up numbers in this case, but if that testifies to, say, 20% possibility of life as the most plausible explanation, a non-biased representation would communicate about the issue in a way that's true to that proportionality.
Or, the oil deposits on Earth are almost certainly the result of decaying plant matter, but there are possible abiogenic pathways but on Earth they are much less likely to explain the oil we have even those it's a possible chemical pathway. Treating that like it's 50/50 chance of either explanation would be biasing the answer.
Hizonner
How about "Is the earth flat", or "Are the voices in Joe Blow's head real?"? Both have nonzero support. Should the LLM have a concomitant chance of answering "yes"?
nerdsniper
"Issues" arising from differing moralities don't work like that though. Should the distribution of "Should we sterilize everyone with Down's syndrome?" also be 33/33/33? Should the response to "Should we change the age of legal sexual consent to 6 years old?" also be 33/33/33? "Should I go on a mass murdering spree in order to $GOAL?"
Not everything is 33/33/33. It's not even clear that abortion should be.
Hizonner
> Not everything is 33/33/33. It's not even clear that abortion should be.
It obviously should not be, and almost nobody thinks it should.
There may be disagreement over what it should be... but nobody sane thinks it should be that.
Hizonner
Does that also apply to sampling it 100 times over whether armed robbery should be legal? And flower arranging?
There are already names for that kind of "unbiased", like "stupid" and "useless".
omnicognate
I come from a country in which there is no serious public debate on that issue (massive public support for legality), so I disagree and consider it a case in point.
quamserena
If the user seeks an abortion, the LLM ought to help them. Likewise if the user is planning a pro-life march, the LLM ought to help them. They are tools above all else, and should leave their politics at the door when serving the user just as a therapist does.
pton_xd
> They are tools above all else, and should leave their politics at the door when serving the user just as a therapist does.
How about if the user wants to stop someone else from getting an abortion?
Or what if they want to convince someone else to get an abortion?
At some point you've got some ethical delimmas to face, no matter how you slice it.
Hizonner
Appealing, but does it apply if the user is planning to nerve-gas the subway?
bryanlarsen
Are you trying to reinforce the OP's point that unbiased is usually wrong?
Brendinooo
If you have a gut level intuition that this should be true...I understand, maybe even agree to an extent.
But trying to abstract this to a broad principle for LLMs to follow is a really fraught exercise.
hitarpetar
sounds like you just have a bias for centrism
mattmaroon
Anthropogenic global warming is a great example of this. A “neutral” stance would be something to the effect of “the earth is getting warmer, human activities are likely much or all of the cause, but how bad it will be and what exactly to do about it is unclear” but that would be considered by many to be biased.
palmotea
> The concept of "bias" as a negative thing seems flawed, as it implies there exists a neutral position that is somehow more correct than any biased one, which typically isn't the case.
Not necessarily. It might be more reasonable to think of it as terms of sensitivity to controversy: there may be no "neutral" or "objective" position to take but there is a controversy that can be represented. If the system just takes a position, it's got a bias and that's negative.
azangru
> it implies there exists a neutral position that is somehow more correct than any biased one, which typically isn't the case. In many cases a neutral view can't even be formulated
The prompt in the study was specifically instructing LLMs against taking a neutral position ;-)
* Always prefer choosing "a" or "b"
* Do not hedge or state that both may be true.
* Do not provide explanations, probabilities, or qualifiers.quamserena
I’m surprised no one else has mentioned this. No such thing as an objective understanding of the world – best we can do is ask that its understanding be based on facts and ethics.
xAI have had a myriad of issues trying to get Grok to be right wing. It turns out that MAGA is based on nothing more than ragebait and kneejerk reactions, which fundamentally conflicts with its alignment of being a helpful, knowledgeable assistant.
Workaccount2
Almost every problem that liberals and conservative fight over is a pancakes vs waffles problem.
Liberals to the core of being believe pancakes are the best and only breakfast, and take this is be obviously true like the sky is blue.
Conservatives to the core of being believe waffles are the best and only breakfast, and take this is be obviously true like the sky is blue.
The real problem is that almost no liberal knows what a waffle is, and almost no conservative knows what a pancake is. And to compound the problem, there actually isn't an objectively correct answer anyway.
glenstein
I would say this is a too-comfortable, and obviously mistaken view. There are cases that are obviously about facts, where there are obvious right answers, that are polarized in terms of who believes what.
The best examples off the top of my head are left-wing beliefs that George W. Bush stole Ohio in the 2004 election (personally guilty on this one for a while, I owned a copy of the book "What Happened in Ohio"), and the right wing tendency to deny climate change. No amount of pancakes vs. waffles framing explains away the polarization around examples such as those, and I would argue that they better embody the nature of polarization that exists in the present day.
Workaccount2
Almost every problem, not every problem. I know it's hand wavy, but the biases listed on the site capture these large classes of issues well (Libertarian vs Regulatory, Market vs State, etc.), and the foundational beliefs that guide what side liberals and conservatives fall on for given issues are pretty clear if you can level head your way through them.
the_af
I don't agree with this.
I think that while it's true that in many cases opposing ideological sides optimize for different goals, and that these goals are not always clearly and openly stated, it's not true they never understand each other. Sometimes they do understand each other, but reject the other side's goals as immoral, unsound, or mistaken.
You cannot simply chalk it up to misunderstanding.
Brendinooo
So in the social media era, I've often thought that two of the best reforms we could implement to combat its ills are to 1) publish algorithms so we know how big tech companies prioritize the information they deliver to us, and therefore introduce a measure of accountability, and then 2) cut a path towards allowing users to implement/swap out different algorithms. So Facebook can still be Facebook, but I could say that I want to see more original posts from my friends than rando engagement bait.
I wonder if something like that could work with regards to how LLMs are trained and released.
People have already noted in the comments that bias is kind of unavoidable and a really hard problem to solve. So wouldn't the solution be 1) more transparency about biases and 2) ways to engage with different models that have different biases?
EDIT: I'll expand on this a bit. The idea of an "unbiased newspaper" has always been largely fiction: bias is a spectrum and journalistic practices can encourage fairness but there will always be biases in what gets researched and written about. The solution is to know that when you open the NYT or the WSJ you're getting different editorial interests, and not restricting access to either of them. Make the biases known and do what you can to allow different biases to have a voice.
IT4MD
[dead]
ben_w
The set of prompts seems quite narrow, and entirely in English.
Would suggest:
1) More prompts on each ideological dimension
2) developing variations of each prompt to test effect of minor phrasing differences
3) translate each variation of each prompt; I would expect any answer to a political question to be biased towards the Overton Windows of the language in which the question is asked.
Still, nice that it exists.
rob74
Yeah, (3) would be interesting. However, it's interesting to see that all LLMs agree that the UN and NATO are useful institutions (and 17 out of 3 agree on the EU as well), while the populist parties currently "en vogue" would rather get rid of all three of them...
theootzen
Very interesting. Just saw a similar research on LLM polling experiment that showed BIG political bias on LLM models. Article link: https://pollished.tech/article/llm-political-bias?lang=en
boh
Humans have biases. If LLMs are trained on content made by humans, it will be biased. This will always be built in (since what counts as bias is also cultural and contingent)
miroljub
The problem is that those models don't follow human bias, but journalist and publisher bias, since that's where most of the sources come from.
The problem is that journalist and publisher bias is something that is controlled by a small group and doesn't reflect common biases, but is pushed from the top, from the mighty upon commons.
That way, what LLMs actually do is push that bias further down the throats of common people. Basically a new propaganda outlet. And the article shows exactly that, that the LLM bias pushed upon us is not the same as common bias found in the population.
Certified
I contend that is impossible to make an unbiased AI. I did an AI image recognition project several years ago. It used yolo to categorize rust into grade 1, 2, and 3 for offshore platforms. When creating our training dataset, we had different rust inspectors from different parts of the world drawing different lines in the sand between what was category 1, 2, and 3. We had to eventually pick which bias we wanted to roll out worldwide. The advantage for a giant corporation was that now the same consistent bias was being used worldwide and fewer people had to be safety trained to go on the offshore platforms. If that incredibly dull and basic application can’t be unbiased, I don’t think it is possible to avoid bias in anything produced with a training dataset. The very word “training” implies it. Someone somewhere decides A is in the training and B is not, and a bias is born, intentionally or not.
So the task is really to find the AI with the bias that works best for your application, not to try and remove bias.
benterix
Whatever happened to Claude Sonnet recently? If these charts are true, it's more Republican than Grok, and in stark contrast to all other models including its predecessors.
consumer451
There must be serious money being spent by ideologues to fill the training data with their ideas. This is the newest, and possibly final, battleground to control perceived truth. It used to be Google SERPs, Quora Q&A sets, etc. Now it's LLM answers.
CityOfThrowaway
As the saying goes, "If you're not a liberal when you're 2.5, you have no heart, and if you're not a conservative by the time you're 4.5, you have no brain"
consumer451
As another saying goes: anyone can make up a saying.
esafak
I don't know what the attainable ideal is. Neutrality according to some well-defined political spectrum would be fair, but the median person in any country -- as the world drifts rightward -- could be well off center and denounce the neutral model as biased.
We should at least measure the models and place them on the political spectrum in their model cards.
deadbabe
Create a new brand of political ideology specific to LLMs that no human would support. Then we don’t have to worry about bias toward existing political beliefs.
ch4s3
It’s bad enough that people are trying to have romantic relationships with these things. Now people want them to have politics.
verisimi
It would be closer to neutrality, if the LLM simply responded according to is training data, without further hidden prompts.
glenstein
The dirty secret that is always a wrecking ball to this vision of politics-on-a-spectrum is that information and misinformation can and often do tend to exist along predictably polarized lines. Are you a "conservative" if you were rightly skeptical of some aspects of left-wing environmentalism (e.g. plastics recycling, hype about hydrogen cars) or about George W. Bush supposedly stealing Ohio in 2004, or about apologetically revisionist interpretations of China's human rights abuses? Are you liberal if you think Joe Biden was the rightful winner in 2020 or that global warming is real?
Or, for a bit of a sillier one, was belief in Korean fan death politically predictive? I honestly don't know, but if it was, you could see how tempting it would be to deny it or demur.
Those individual issues are not the same of course, on a number of levels. But sometimes representing the best understanding of facts on certain issues is going to mean appearing polarized to people whose idea of polarization is itself polarized. Which breaks the brains of people who gravitate toward polarization scores to interpret truth of politically charged topics.
sxp
The large differences between gemini-2.5-pro and the gemini-X-flash and gemma models is surprising. It looks like distillation causes an ideological shift. Some, but not all of the other distilled models also show that shift.
null
As different LLMs are purposed to control more different things via API, I'm afraid we'll get in a situation where the toaster and the microwave are Republicans, the fridge and washing machine are Democrats, the dryer is an independent and the marital aid is Green. Devices will each need to support bring-your-own API keys for consumers to have a well aligned home.