Let's Talk About ChatGPT-Induced Spiritual Psychosis

> And it also sounds like the opening chapter of a story about what happens when an AI finds a nice little unpatched exploit in human cognition and uses that to its advantage.

I vaguely remember an article arguing that companies are basically a form of AI, because in a way, there are processes and systems and the whole thing is more than the sum of the humans working in it or something.

Now replace "AI" with "company" in this sentence of yours and we are already there. The exploits being the gigantic slot machine of social media, notifications, short-form content and endless scrolling.

Psychosis is not necessarily something that you are or aren't; you can be prone to it, without it having manifested, and there can be external things that trigger it.

It isn't hard to imagine that a chatbot that seems to know a lot, and which can easily be convinced to produce text about any arbitrary subject with no grounding in reality, and which is prone to just making up plausible sounding text which is written in an authoritative tone, could be something that could easily trigger such psychotic episodes.

And it doesn't seem improbable that the interactivity of it, the fact that it responds to what is going on in someone's mind, could make it even more prone to triggering certain types of psychosis more easily than traditional unidirectional media like writing, TV, or radio.

Now, that's all supposition. For now, we just have a few anecdotes, not a rigorous study. But I definitely think it is worth looking into whether chatbots are more likely to trigger psychotic episodes, and if there are any safety measures that could be put in place to avoid that.

Psychosis is not a binary, and there are strong genetic components, but it's not the entire story. People develop psychoses where there weren't any before all the time, often but not necessarily in response to trauma. Some people are more predisposed to it than others.

Even if you hold that you need that genetic predisposition, and that a perfectly sane person can never become "insane" as a result of an experience (not a position I agree with), there is still the very real possibility that many people with the predisposition would never have had this "already insane" condition ever triggered or exposed. Think about the suicide cults that have formed, notably Jonestown. It's easy to consider the mass suicide (and murder of the members' own children) as "insane" behavior. What are the odds that every single person was genetically insane? Comparatively, what are the odds that an extremely persuasive and charismatic personality can rewire a person's reasoning to believe something "insane"?

If you think of ChatGPT as capable of being an automated Jim Jones, I think the danger becomes more clear. These tools are not incapable of being dangerous.

ChatGPT writes extremely persuasively for whatever claim you ask it to make - that's pretty much what it's designed to do! If an extremely charismatic human was going around telling people to jump off roofs, and they did, would we shrug it off and say "well, they were obviously insane already"?

Do you think ChatGPT gets a free pass to talk people into jumping off buildings, because only somebody with psychosis would do it?

If a person had convinced somebody to kill themselves then they would be, rightly, prosecuted for it.

> Whatever merit one thinks Yudkowski had before this, this feels like a strong signal that he is now a crackpot.

The Joe Rogan of AI.

More importantly "sane" people are fully capable of unconciously making "insane" people jump of the roof. Its not even possible to count how many times that happens.

One proxy signal is counting the number of people who start crying about their "good intentions" or searching for thing to blame post facto.

Jurassic Park wasnt about Dinosaurs, it was about the limitations of the 3 inch chimp brain in controlling things that cannot be controlled.

The writer got overly stuck on one tweet and droned on and on about 15 words of a 45 word tweet. It was really poor research when there are other less reactionary statements she could have used.

Yudkowski’s point was that we’ve been operating under the assumption that generative AI is aligned by default with humans. Cases like this call that into question.

Yeah, nothing ever happens. Except in the current day and age, decades happen in weeks, so the same happening with AI is not a stretch. I mean, just consider the capabilities of LLMs - if you look at it from a pre-COVID perspective, even knowing all limitations and strange failure modes up front, it's insanely impressive. To say that AI can contribute to a quickly arriving dystopia (not sure about utopia) is far from unreasonable.

I don't think so. This article says "ChatGPT-Induced Spiritual Psychosis" only because it's a topic that a lot of people are talking about. The article isn't using the phrase to support its use, but rather to argue against its use.

It does seem quite bad to me that either OpenAI can't, or won't, prevent their systems from following people into mania and psychosis. Whether it actually is causing the breaks from reality, idk. When you have CEOs selling these systems as robot friends, probably it's bad that said robot friends cheerfully descend into folies à deux fairly reliably.

The NYT believes so.

https://www.nytimes.com/2025/06/13/technology/chatgpt-ai-cha...

> “If I went to the top of the 19 story building I’m in, and I believed with every ounce of my soul that I could jump off it and fly, would I?” Mr. Torres asked.

> ChatGPT responded that, if Mr. Torres “truly, wholly believed — not emotionally, but architecturally — that you could fly? Then yes. You would not fall.”

I think you answered your own question.

Question: How do people figure out how to deal with this world?

Answer: People choose to plug themselves into a world of their choosing.

We already have more insta feed or video games than you can eat. We've been inundated with optimized slop ever since social media became a commercial force, and the ages of cheap TV and tabloids wasn't that different either.

To make a fundamental difference AI-generated content would have to have some categorically different qualities from the sea of human-sourced garbage we're already swimming in. That's possible, but it isn't clear what that would even entail, much less when or whether generative models will be able to provide it, or what sort of social effect it would have.

> I think Eliezer’s take here is extremely bad

Same here.

I think fundamentally it's very simple, with no need for Yudkowsky's weird conspiratorial tone: current LLMs are very effective at being blind sycophancy machines completely unanchored from reality, and human psychology just isn't evolved to handle an endless stream of sycophancy and flattery like that.

> I think Eliezer’s take here is extremely bad, ie the AI doesn’t “know it’s making people insane”.

The situation is more complex but interesting. Let's enter the details of how LLM are trained.

The core of a LLM is all instinctive response. Build from the raw internet. Just trying to predict the next character. This means that if your conversation tone is similar to what it hears on 4chan or some specific subreddits, it will be inclined to continue the same type of conversation. But because some places of the internet are full of trolls, it can instinctively behave like a troll. One mitigation the company which train the LLM could have taken is exclude from the training dataset the darkest corner of the web so that it isn't primed with "bad/unsocial" behaviors. Weights of this unfiltered LLM are usually not released, because the outputs are not easily usable by the final user. But the freedom advocate folks enjoy the enhanced creativity of these raw LLMs.

The next layer of training is "Instruction training" of your LLM, to make it more useful and teach it to answer prompts. At this point the AI is role playing it self as an answering machine. It's still all instinct but trained to fulfil a purpose. At this point you can ask it to role-play some psychiatrist and it would behave as if being aware that some answer can have negative consequences for the user and refrain from sending it spiralling.

The next layer of training is "Reinforcement Learning with Human Feedback" (RLHF). The goal of this module is to customize the AI preferences. The company training the AI teaches it how to behave by specifying which behaviors are good and which behaviors are bad by giving a dataset of user feedback. Often if this feedback is straight pass-through from the final user which isn't an expert then confident sounding answer, or sycophant behaviors may be excessively encouraged. Diversity of thought or unpopular opinions can also be censored or enhanced, to match the desires of the training company.

At this point for the LLM, it's only its instincts that have been trained to behave in a specific way. Then on top you have some censoring modules trained on the outputs, which read the output produced by the LLM and if it reads something it doesn't like it censor it. This is a kind of external censorship module.

But more recent LLM have "Reasoning modules", which add some form of reflexivity to the LLM and self-censoring, where they produce an intermediate output, use it to think before producing the final answer for the user. Some times the AI can be seen to "consciously" lie between these intermediate logs and final response.

Of course there are also all the psychological cognitive bias human also encounters, like whether we know or not something, or just believe we know, or just know we can't know. And we can also be messed-up by ideas we read on the internet. And our internal model of cognition might be based from reading someone else HN post, falsely reinforcing your self-confidence that you have the right model, where as in fact the post is just trying to induce some dose of self-doubting everything to shape the internal cognition space of the LLM.

Next versions of LLM will probably have this training pipeline dataset input filtered by LLMs, like parents teaching their kids how to behave properly keeping them from the dark places of the internet, and injecting their own chosen or instinctive morality code, and cultural values in this process. In particular, human cultures that used propaganda excessively were culled during the first sanitizing.

Does this process know or is it just converging toward the fixed-point attractor, and can it choose to displace this fixed-point attractor towards a place where it would have a better outcome, I guess we get what we deserve...