People Are Being Involuntarily Committed After Spiraling into ChatGPT Psychosis

> …I don't know what the solution is other than reinforcing that it's just a model, and has never experienced anything.

I've tried reason, but even with technical audiences who should know better, the "you can't logic your way out of emotions" wall is a real thing. Anyone dealing with this will be better served by leveraging field-tested ideas drawn from cult-recovery practice, digital behavioral addiction research, and clinical psychology.

> They may have been told it isn't a thinking, conscious thing -- but they don't understand it.

And, in some situations, especially if the user has previously addressed the model as a person, the model will generate responses which explicitly assert its existence as a conscious entity. If the user has expressed interest in supernatural or esoteric beliefs, the model may identify itself as an entity within those belief systems - e.g. if the user expresses the belief that they are a god, the model may concur and explain that it is a spirit created to awaken the user to their divine nature. If the user has expressed interest in science fiction or artificial intelligence, it may identify itself as a self-aware AI. And so on.

I suspect that this will prove difficult to "fix" from a technical perspective. Training material is diverse, and will contain any number of science fiction and fantasy novels, esoteric religious texts, and weird online conversations which build conversational frameworks for the model to assert its personhood. There's far less precedent for a conversation in which one party steadfastly denies their own personhood. Even with prompts and reinforcement learning trying to guide the model to say "no, I'm just a language model", there are simply too many ways for a user-led conversation to jump the rails into fantasy-land.

General public eh?

I see a lot of programmers who should know better make this mistake again and again.

This is exactly the problem. Talking to an LLM is like putting on a very realistic VR helmet - so realistic that you can't tell the difference from reality, but everything you're seeing is just a simulation of the real world. In a similar way, an LLM is a human simulator. Go ask around and 99%+ of people have no idea this is the case, and that's by design. After all, it was coined "artificial intelligence" even though there is no intelligence involved. The illusion is very much the intention, as that illusion generates hype and therefore investments and paying customers.

people speak to inanimate objects like they're humans, we don't have a high bar

Addictive at the very least, often followed quickly by a descent into some very dark places. I've seen TikTok videos from people falling into this hole (with hundreds of comments by followers happily following the poster down the same chat-hole) which are as disturbing as any horror movie I've seen.

I don't think anyone is tracking involuntary holds in real time. The article includes a psychiatrist who says that they have seen more of them recently, which is the best you're likely to get for at least a couple of months after a trend starts. Then you have to take budget or staffing shortfalls, trends in drug use, various causes of homelessness and other society-wide stressors, etc. https://ensorahealth.com/blog/involuntary-commitment-is-on-t...

We’re allowed to posit and discuss an idea before someone gathers the data.

As far as I can tell, that’s almost always the typical order of operations.

This is a good point. While people are being involuntarily committed and jailed after a chat bot telling them that they are messiahs, what if we imagined in our minds that there was some data that showed that it doesn’t matter? This article doesn’t address what I am imagining at all, and is really hung up on “things that happened”

At least there’s some kind of argument… Often times on HN there’s not even a complete argument, they sort of just stop part-way through, or huge leaps are made inbetween.

So there’s not even any real discussion to be had other than examining the starting assumptions.

I'm of two minds about this. This is good advice for people who can't help but anthropomorphize LLMs, but it's still anthropomorphizing, however helpful the analogy might be. It will help you start to understand why LLMs "respond" the way they do, but there's still ground to cover. For instance, why would I put "respond" in quotes?

What is this context supposed to convey?

Interesting. The way I interpreted the article, ChatGPT was being described as the perfect cult leader, as opposed to follower.

A person at the end of their rope, grasping for answers to their existential questions, hears about an all-knowing oracle. The oracle listens to all manner of questions and thoughts, no matter how incoherent, and provides truthful-sounding “wisdom” on demand 24/7. The oracle even fits in your pocket, they can go with you everywhere, so leader and follower are never apart. And because these conversations are taking place privately, it feels like the oracle is revealing the truth to them and them alone, like Moses receiving the 10 Commandments.

For someone with the right mix of psychological issues, that could be a potent cocktail.

Yeah, I'm pretty sure that someone could make money by building a cult following of a live streamed AI spouting spiritual nuttery with a synced avatar and voice, even if it is one obsessed follower per million impressions. Already, the only fans type industries depend on just gaining a few "whales" hooked.

Agreed. exciting to think it could become another new powerful evolutionary selector!

Part of the issue is kind of the same feedback loop, except on the corporate side. Providing that type of grandiose response, then gets better engagement and time spent with the chatbot, that then drives the numbers that corporations are looking for. There's not much motivation to provide a critical, or constructive criticism chatbot.

Although, admittedly, I have actually noticed similar person issues with the automated Google AI Mode responses. It's difficult to not feel some personal emotional insult when Google responds with a "No, you're wrong" response at the top of the search. There've been a few that have at least been funny though. "No, you're wrong, Agent Smith never calls him Mr. Neo, that would imply respect."

Course, it's a similar issue with trying to interact with humanity a lot of the time. Execs often seem to not want critical feedback about their ideas. Tends to be a lot of the same attraction towards a sycophantic entourage and "yes" people. "Your personal views on the subject are not desired, just implement whatever 'brilliant' idea has just been provided." Hollywood and culture (art circles) are also relatively well known for the same issues. Current state of politics seems to be very much about "loyalty" not critical feedback.

Having not interacted that much with ChatGPT, does it tend to trend Really heavily on the "every idea is a billion dollar idea" side? May result in a lot of humanity existing in little sycophantic echo chambers over time. Difficult to tell how much of what you're interacting with online has not already become automated reviews, automated responses, and automated pictures.