Facts don't change minds, structure does

I remember reading an article on one of the classic rationalist blogs (but they write SO MUCH I can't possibly find it) describing something like "rational epistemic skepticism" – or maybe a better term I can't recall either. (As noted below: "Epistemic learned helplessness")

The basic idea: an average person can easily be intellectually overwhelmed by a clever person (maybe the person is smarter, or more educated, or maybe they just studied up on a subject a lot). They basically know this... and also know that it's not because the clever person is always right. Because there's lots of these people, and not every clever person thinks the same thing, so they obviously can't all be right. But the average person (average with respect to whatever subject) is still rational and isn't going to let their beliefs bounce around. So they develop a defensive stance, a resistance to being convinced. And it's right that they do!

If someone confronts you with the PERFECT ARGUMENT, is it because the argument is true and revelatory? Or does it involve some slight of hand? The latter is much more likely

The best way to lie is not presenting false facts, it's curating facts to suit your narrative. It's also often that you accidentally lie to yourself or others in this way. See a great many news stories.

To add to your second point, those algorithms are extremely easy to game by states with the resources and desire to craft narratives. Specifically Russia and China.

There has actually been a pretty monumental shift in Russian election meddling tactics in the last 8 years. Previously we had the troll army, in which the primary operating tactic of their bot farms were to pose as Americans (as well as Poles, Czechs, Moldovans, Ukrainians, Brits, etc.) but push Russian propaganda. Those bot farms were fairly easy to spot and ban, and there was a ton of focus on it after the 2016 election, so that strategy was short lived.

Since then, Russia has shifted a lot closer to Chinese style tactics, and now have a "goblin" army (contrasted with their troll army). This group no longer pushes the narratives themselves, but rather uses seemingly mindless engagement interactions like scrolling, upvoting, clicking on comments, replying to comments with LLMs, etc., in order to game what the social media algorithms show people. They merely push the narratives of actual Americans (not easily bannable bots) who happen to push views that are either in line with Russian propaganda, or rhetoric that Russian intelligence views as being harmful to the US. These techniques work spectacularly well for two reasons: the dopamine boost to users who say abominable shit as a way of encouraging them to do more, and as a morale-killer to people who might oppose such abominable shit but see how "popular" it is.

https://www.bruegel.org/first-glance/russian-internet-outage...

Thanks for your thoughts, they perfectly extend mine. I agree that it would be a sign of a very fragile belief system if it gets unwound by a single bit of contradictory evidence. And as to the "facts" that we're getting 24/7 coming out of every microwave is just a sign of complete decoupling of people's beliefs from empirical reality, in my humble opinion. Supply and demand and all that.

> If you believe in climate change and encounter a situation where a group of scientists were proven to have falsified data in a paper on climate change, it really isn't enough information to change your belief in climate change, because the evidence of climate change is much larger than any single paper.

Although your wider point is sound that specific example should undermine your belief quite significantly if you're a rational person.

1. It's a group of scientists and their work was reviewed, so they are probably all dishonest.

2. They did it because they expected it to work.

3. If they expected it to work it's likely that they did it before and got away with it, or saw others getting away with it, or both.

4. If there's a culture of people falsifying data and getting away with it, that means there's very likely to be more than one paper with falsified data. Possibly many such papers. After all, the authors have probably authored papers previously and those are all now in doubt too, even if fraud can't be trivially proven in every case.

5. Scientists often take data found in papers at face value. That's why so many claims are only found to not replicate years or decades after they were published. Scientists also build on each other's data. Therefore, there are likely to not only be undetected fraudulent papers, but also many papers that aren't directly fraudulent but build on them without the problem being detected.

6. Therefore, it's likely the evidence base is not as robust as previously believed.

7. Therefore, your belief in the likelihood of their claims being true should be lowered.

In reality how much you should update your belief will depend on things like how the fraud was discovered, whether there were any penalties, and whether the scientists showed contrition. If the fraud was discovered by people outside of the field, nothing happened to the miscreants and the scientists didn't care that they got caught, the amount you should update your belief should be much larger than if they were swiftly detected by robust systems, punished severely and showed genuine regret afterwards.

The idea that people believe in climate change (or evolution) is odd considering people don't say they believe in General Relativity or atomic theory of chemistry. They just accept those as the best explanations for the evidence we have. But because climate change and evolution run counter to some people's values (often religious but also financially motivated), they get called beliefs.

> the previous era of corporate controlled news media... The facts you are exposed to today are usually decided by an algorithm

... But that algorithm is still corporate controlled.

See also: the Chinese robber fallacy.

Even if only 0.1% of Chinese people engaged in theft, and that would be a much lower rate than in any developed country, you'd still get a million Chinese thieves. You could show a new one every day, bombarding people with images and news reports of how untrustworthy Chinese people are. The news reports themselves wouldn't even be misinformation, as all the people shown would actually be guilty of the crimes they were accused of. Nevertheless, people would draw the wrong conclusion.

Wow. I'm reminded of a great essay/blgo I read years ago that I'll never find again that said a good, engaging talk/presentation has to have an element of surprise. More specifically, you start with an exposition of what your audience already knows/believes, then you introduce your thesis which is SURPRISING in terms of what they already know. Not too out of the realm of belief, but just enough.

The bigger/more thought-diverse the audience, the harder this is to do.

I completely agree with your comment. The common narrative about Galileo and the Church is often oversimplified and overlooks the intellectual context of the time. As you pointed out, it wasn’t about a crude Biblical literalism—after all, even centuries before Galileo, figures like Saint Thomas Aquinas, drawing on Aristotle, already accepted that the Earth is spherical.

By Galileo’s era, the Catholic Church was well aware of this scientific truth and actively engaged with astronomy and natural philosophy. The dispute was far more about competing models and the standards of evidence required, not a refusal to accept reason or observation.

Then I can’t help but think: if the author of the article didn’t even understand this, how can the rest of the article be correct if it started from a biased and almost false premise?

> the case between Galileo and the church was way more nuanced than is popularly retold

Ex historian here. This is true. It’s a complicated episode and its interpretation is made more murky by generations of people trying to use it to make a particular rhetorical point. Paul Feyerabend is guilty of this too, although he’s at least being very original in the contrarian philosophy of science he’s using it for.

If anyone is interested in the episode for its own sake (which is rare actually, unless you’re a renaissance history buff first and foremost), I’d probably recommend John Heilbron’s biography which has a pretty balanced take on the whole thing.

I just recently watched a lecture about this and was fascinated.

Specifically, the (incorrect) model of the universe that was used in Europe at the time had been refined to the point that it was absurdly accurate. Even had they adopted a heliocentric model, there would have been no direct benefit for for a long, long time. If anything, Galileo's work was rife with errors and mathematical problems that would have taken a lot of work to figure out.

So the argument was to take on a bunch of technical debt and switching costs for almost no benefits.

Does Feyerabend explain why Galileo was placed under house arrest?

Perhaps I'm missing some nuance here, but I don't see why a rational argument about competing models would require such drastic suppression.

Reminds me of the Galileo podcast series in the Opinionated History of Mathematics by Viktor Blasjo: https://intellectualmathematics.com/opinionated-history-of-m...

> and had nothing whatsoever to do with Biblical literalism like the passage in Joshua about making the sun stand still.

The church is and was a large, often heterogenous institution. For some the issue was about conflict with literal interpretations of the bible, not merely the predominate allegorical interpretations (a more widely held concern, at least as a pedagogic matter). AFAIU, while the pope wasn't of this mind, some of the clerics tapped to investigate were. See, e.g., the 1616 Consultant's Report,

> All said that this proposition is foolish and absurd in philosophy, and formally heretical since it explicitly contradicts in many places the sense of Holy Scripture, according to the literal meaning of the words and according to the common interpretation and understanding of the Holy Fathers and the doctors of theology.

https://www.vaticanobservatory.org/sacred-space-astronomy/in...

As I’ve grown older and witnessed history in action I’ve begun to understand that reality is much, much more complicated than the simple narratives of history we lean on as a society.

Just think of how many different competing narratives are currently in existence surrounding this tumultuous point in history and realize that at some point some of these narratives will become dominant. Over time as the events leave social memory the key conclusions will likely be remembered but a lot of the reasoning behind them will not. As it exits living memory most of the nuance and context is lost. Over time we may change the narrative by reconsidering aspects that were forgotten, recontextualizing events based on modern concepts and concerns, misunderstanding what happened, or even surreptitiously “modifying” what happened for political ends. Or to put it more plainly, history is written by the victors and can be rewritten as time goes on and the victors change.

The important thing is not why they thought they were right but the fact they could not tolerate being wrong, or even tolerate dissidence on that one little inconsequential thing.

That's why you have people today pushing for flat earth and creationism.

Because their whole shtick is we are always right about absolutely everything.

It's also noteworthy that she was willing to sit with and listen to them in the first place.

AI chat bots in the future may be a part of ritual mind cleansing.

I think this is just gloating. Children leaving home for college and quickly abandoning the belief systems of their family is almost more common than the opposite, where they maintain them. Especially if the belief system is something as unpopular as white supremacy mythology; not easy to make new friends at your new school if you don't give that up.

I'm sure she maintains many beliefs that may people would see as racist, along with her classmates. She hasn't been educated or fixed, she just left home.

I remember my first year in college as being the time when I solidified my own first worldview. Prior to that, I had some ideas like the existence of God (in some form) that I was ambivalent about or maybe deferring final judgement. That's when I decided that I was an atheist.

Coincidentally, around the same time my twin brother became a serious Christian. He was socially integrated into a group. He finished college. I did not.

Then years later, maybe late 20s or early 30s, I became convinced that I had been wrong about my government my whole life and that they were not trustworthy. 9/11 being a false flag (which I still believe) was evidence of that.

The interesting thing was at the time when I was in New York I had completely accepted the idea that those three buildings had all turned into dust because the jet hit them. I remember walking around lower Manhattan to pick up a check and the dust was just coating everything.

I had even done some word processing on one of the twin towers leases shortly before the event while temping at Wachtell Lipton. At the time I made no connection.

Anyway, I think an underappreciated aspect of belief graphs is their connection to social groups and identity. It was much easier for me to question institutions when I already felt more marginalized and actually partly blamed society for it being so hard for me to handle my needs and find a place in it.

Another aspect of group membership and beliefs is practical. When groups are competing strategically, they often do so in ways that are not particularly ethical. It's much easier to justify this if you think of the other group as being deeply flawed, evil, invaders, etc.

Although some of these demonization s of the other group do have some kernel of truth to them, they are largely oversimplifications in the belief graphs leading to dangerous inaccuracies.

What are the practical structural and cultural differences that lead to the group divisions? They largely seem geographic, economic, ethnic.

Could a more sophisticated, better integrated, and more accurate belief system help? Or do the social structures and networks largely define the groups?

Are we just basically mammalian ant colonies? Brutally fighting each other for dominance any time there is a resource conflict?

If the other side seems to be trying to hog important resources any time they get a chance, you perceive that you are not playing a fair game. It's not a civil interaction. The other doesn't play by the rules or tell the truth or leave any subtly in discourse. So why should your group, unless it wants to get wiped out?

In my worldview the faint hope is that having more abundance of resources will somehow lead to more civility.

I really appreciate this feedback, I'll look into both of those before the next post. Just wanted to say thanks.

Just to add to this: I couldn’t read the text in the white boxes of the graphs. Very unfortunate choices of colors…

The edges are labeled by transitive verbs, where the arrow points from the subject of that verb to the direct object. (I'm counting particle verbs, like "leads to", as verbs.) The nodes are labeled by nouns. If you can change a noun to a verb, I guess you would be changing what is an edge and what is a node.

Example: In the article's first diagram, there is a node labeled "Innovation". This could be replaced by a node labeled "Capitalist" and a node labeled "Improvement", with an arrow from the first to the second labeled "innovates."

So yes, if you can replace a node by an edge (and vice versa, although I don't give an example), this node vs. edge thing is fuzzy.

I just really wish most people had this same kind of attitude, but can't in good faith say that's what I'm observing.

I've really enjoyed Haidt's book, though its really a couple of different books in one. I need to read his other work.

To your point about left and right, an interesting point I heard recently is that the left is coalition-driven whereas the right is consensus-driven (at least in US politics). Mapping this back to Haidt, one of his findings is that the left tends to greatly emphasize one or two of the "moral taste receptors", with the right having a roughly equal emphasis between them. It isn't clear to me how these two points might explain each other, but I do wonder if there isn't some self-reinforcement there. If there is, I wonder how/if that might explain political systems more widely.

>>The reason the political right is so cohesive (vs the left)

Citation excruciatingly needed. This feels like recency bias imo. The Right (I'm assuming we're going US here?) is a coalition of people all walks just as much as the left. I mean, right now large chunks Trump voters are rioting over the Epstein non-release and all the people who were in it for the tax breaks are trying to convince them to stop.

Interesting. I often hear/read the term "steelmanning" instead, as in the opposite of constructing a straw man argument.