AI therapy bots fuel delusions and give dangerous advice, Stanford study finds
139 comments
·July 13, 2025m3047
BLKNSLVR
I'm just going to re-write what you've written with a bit of extra salt:
Artificial intelligence: An unregulated industry built using advice from the internet curated by the cheapest resources we could find.
What can we mitigate your responsibility for this morning?
I've had AI provide answers verbatim from a self-promotion card of the product I was querying as if it was a review of the product. I don't want to chance a therapy bot quoting a single source that, whilst it may be adjacent to the problem needing to be addressed, could be wildly inappropriate or incorrect due to the sensitivities inherent where therapy is required.
(likely different sets of weightings for therapy related content, but I'm not going to be an early adopter for my loved ones - barring everything else failing)
theendisney
My theory is that the further up the hierarcy the beneficial decisions are often harmful to those below which requires emotional distancing which even further up becomes full blown collective psychopaty. The yes men grow close while everyone else floats away.
vasco
> As someone who is therefore historically familiar with this process in a wider systemic sense
What does "being historically familiar with a process in a wider systemic sense" mean? I'm trying to parse this sentence without success.
kelseyfrog
I'm reading it to say, having working knowledge of intra-personal structures in a way that is contingent on historical context. These would be social, economic, religious, family, political, patterns of relation that groups of people exist in.
The assumption GP is making is that the incentives, values, and biases impressed upon folks providing RL training data may systematically favor responses along a certain vector that is the sum of these influences in a way that doesn't cancel out because the sample isn't representative. The economic dimension for example is particularly difficult to unbias because the sample creates the dataset as an integral part of their job. The converse would be collecting RL training data from people outside of the context of work.
While that it may not be feasible or even possible to counter, that difficulty or impossibility doesn't resolve the issue of bias.
mlinhares
Every single empire falls into this, right? The king surrounds himself with useless sycophants that can't produce anything but are very good at flattering him, he eventually leads the empire to ruin, revolution happens, the cycle starts anew.
I wish I could see hope in the use of LLMs but i don't think the genie goes back into the bottle, the people prone to this kind of delusion will just dig a hole and go deep until they find the willpower or someone on the outside to pull them out. Feels to me like gambling, there's no power that will block gambling apps due to the amount of money they fuel into lobbying so the best we can do is try to help our friends and family and prevent them from being sucked into it.
dragontamer
Certainly not the story of, ex: the Mongol Empire. Which is the Great Khan dies but he was the big personality holding everything together.
There were competent kings and competent Empires.
Indeed, it's tough to decide where the Roman Empire really began it's decline. It's not a singular event but a centuries long decline. Same with the Spanish Empire and English Empire.
Indeed, the English Empire may have collapsed but that's mostly because Britain just got bored of it. There's no traditional collapse for the breakup of the British Empire
---------
I can think of some dramatic changes as well. The fall of the Tokugawa Shogunate of Japan wasn't due to incompetence, but instead the culture shock of a full iron battleship from USA visiting Japan when they were still a swords and samurai culture. This broke the Japanese trust in the Samurai system and led to a violent revolution resulting in incredible industrialization. But I don't think the Tokugawa Shogunate was ever considered especially corrupt or incompetent.
---------
Now that being said: Dictators fall into the dictator trap. A bad king who becomes a narcissist and dictator will fall under the pattern you describe. But that doesn't really happen all that often. That's why it's so memorable when it DOES happen
somenameforme
> the English Empire may have collapsed but that's mostly because Britain just got bored of it. There's no traditional collapse for the breakup of the British Empire
I completely agree with the point you're making, but this part is simply incorrect. The British Empire essentially bankrupted itself during WW2, and much of its empire was made up of money losing territories. This led them to start 'liberating' these territories en masse which essentially signaled the end of the British Empire.
zpeti
How about all the people out there who are at rock bottom, or have major issues, are not leaders, are not at the top of their game, and need some encouragement or understanding?
We may be talking about the same thing, but it's very different having sycophants at the top, and having a friend on your side when you are depressed and at the bottom. Yet both of them might do the same thing. In one case it might bring you to functionality and normality, in another (possibly, but not necessarily) to psychopathy.
AdieuToLogic
How is anyone versed in LLM technical details surprised by this?
They are very useful algorithms which solve for document generation. That's it.
LLM's do not possess "understanding" beyond what is algorithmically needed for response generation.
LLM's do not possess shared experiences people have in order to potentially relate to others in therapy sessions as LLM's are not people.
LLM's do not possess professional experience needed for successful therapy, such as knowing when to not say something as LLM's are not people.
In short, LLM's are not people.
coliveira
Computer scientists are, in part, responsible for the public confusion about what LLMs are and can do. Tech investors and founders, however, are the biggest liars and BS peddlers when they go around saying irresponsible things like LLMs are on the verge of becoming "conscious" and other unfounded and impossible things (for LLMs). It's not a surprise that many people believe that you can have a personal "conversation" with a tool that generates text based on statistical analysis of previous data.
gonzobonzo
It feels like 95% of the people are responding to the headline instead of reading the article. From the article:
> The Stanford research tested controlled scenarios rather than real-world therapy conversations, and the study did not examine potential benefits of AI-assisted therapy or cases where people have reported positive experiences with chatbots for mental health support. In an earlier study, researchers from King's College and Harvard Medical School interviewed 19 participants who used generative AI chatbots for mental health and found reports of high engagement and positive impacts, including improved relationships and healing from trauma. ** > "This isn't simply 'LLMs for therapy is bad,' but it's asking us to think critically about the role of LLMs in therapy," Haber told the Stanford Report, which publicizes the university's research. "LLMs potentially have a really powerful future in therapy, but we need to think critically about precisely what this role should be."
wongarsu
Self-help books help people (at least sometimes). In an ideal world an LLM could be like the ultimate self-help book, dispensing the advice and anecdotes you need in your current situation. It doesn't need to be human to be beneficial. And at least from first-order principles it's not at all obvious that they are more harmful than helpful. To me it appears that most of the harm is in the overly affirming sycophant personally most of them are trained into, which is not a necessary or even natural feature of LLMs at all
Not that the study wouldn't be valuable even if it was obvious
Retric
Self-help books are designed to sell, they’re not particularly useful on their own.
LLM’s are plagued by poor accuracy so they preform terribly in any situation where inaccuracies have serious downsides and there is no process validating the output. This is a theoretical limitation of the underlying technology, not something better training can fix.
Dylan16807
I don't think that argument is solid enough. "serious downsides" doesn't always mean "perform terribly".
Most unfixable flaws can be worked around with enough effort and skill.
intended
I have stopped using an incredibly benign bot that I wrote, even thought it was supremely useful - because it was eerily good at saying things that “felt” right.
Self help books do not contort to the reader. Self help books are laborious to create, and the author will always be expressing a world model. This guarantees that readers will find chapters and ideas that do not mesh with their thoughts.
LLMs are not static tools, and they will build off of the context they are provided, sycophancy or not.
If you are manic, and want to be reassured that you will be winning that lottery - the LLM will go ahead and do so. If you are hurting, and you ask for a stream of words to soothe you, you can find them in LLMs.
If someone is delusional, LLMs will (and have already) reinforced those delusions.
Mental health is a world where the average/median human understanding is bad, and even counter productive. LLMs are massive risks here.
They are 100% going to proliferate - for many people, getting something to soothe their heart and soul, is more than they already have in life. I can see swathes of people having better interactions with LLMs, than they do with people in their own lives.
quoting from the article:
> In an earlier study, researchers from King's College and Harvard Medical School interviewed 19 participants who used generative AI chatbots for mental health and found reports of high engagement and positive impacts, including improved relationships and healing from trauma.
null
never_inline
1. Ars Technica's (OP website) audience includes tech enthusiast people who don't necessarily have a mental model of LLMs, instruction tuning or RLHF.
2. why would this "study" exist? - for the reason computer science academics conduct study on whether LLMs are empirically helpful in software engineering. (The therapy industrial complex would also have some reasons to sponsor this kind of a research, unlike SWE productivity studies where the incentive is usually the opposite.)
AdieuToLogic
Both great points.
For the record, my initial question was more rhetorical in nature, but I am glad you took the time to share your thoughts as it gave me (and hopefully others) perspectives to think about.
mise_en_place
The average person in 2025 has been so thoroughly stripped of their humanity, that even a simulacrum of a human is enough.
BeetleB
> In short, LLM's are not people.
Not really sure that is relevant in the context of therapy.
> LLM's do not possess shared experiences people have in order to potentially relate to others in therapy sessions as LLM's are not people.
Licensed therapists need not possess a lot of shared experiences to effectively help people.
> LLM's do not possess professional experience needed for successful therapy, such as knowing when to not say something as LLM's are not people.
Most people do not either. That an LLM is not a person doesn't seem particularly notable or relevant here.
Your comment is really saying:
"You need to be a person to have the skills/ability to do therapy"
That's a bold statement.
padolsey
>> LLM's do not possess professional experience needed for successful therapy, such as knowing when to not say something as LLM's are not people.
> Most people do not either. That an LLM is not a person doesn't seem particularly notable or relevant here.
Of relevance I think: LLMs by their nature will often keep talking. They are functions that cannot return null. They have a hard time not using up tokens. Humans however can sit and listen and partake in reflection without using so many words. To use the words of the parent comment: trained humans have the pronounced ability to _not_ say something.
BeetleB
All it takes is a modulator that controls whether to let the LLM text through the proverbial mouth or not.
(Of course, finding the right time/occasion to modulate it is the real challenge).
zpeti
A lot of the comparisons I see revolve around comparing a perfect therapist to an LLM. This isn't the best comparison, because I've been to 4 different therapists over my life an only one of them actually helped me (2 of them spent most of the therapy telling me stories about themselves. These are licensed therapists!!) There are really bad therapists out there.
An LLM, especially chatgpt is like a friend who's on your side, who DOES encourage you and takes your perspective every time. I think this is still a step up from loneliness.
And a final point, ultimately an LLM is a statistical machine that takes the most likely response to your issues based on an insane amount of human data. Therefore it is very likely to actually make some pretty good calls about what it should respond, you might even say it takes the best (or most common) in humanity and reflects that to you. This also might be better than a therapist, who could easily just view your sitation through their own live's lense, which is suboptimal.
ClumsyPilot
> You need to be a person to have the skills
Generally a non-person doesn’t have skills, it’s a pretty likely to be true statement even if made on a random subject.
BeetleB
Once again: The argument appears to be "LLMs cannot be therapists because they are LLMs." Circular logic.
> Generally a non-person doesn’t have skills,
A semantic argument isn't helpful. A chess grandmaster has a lot of skill. A computer doesn't (according to you). Yet, the computer can beat the grandmaster pretty much every time. Does it matter that the computer had no skill, and the grandmaster did?
That they don't have "skill" does not seem particularly notable in this context. It doesn't help answer "Is it possible to get better therapy from an LLM than from a licensed therapist?"
ddp26
What's the base rate of human therapists giving dangerous advice? Whole schools, e.g. psychotherapy, are possibly net dangerous.
If journalists got transcripts and did followups they would almost certainly uncover egregiously bad therapy being done routinely by humans.
Eextra953
Therapist have professional standards that include a graduate degree and 1000's of hours of practice with supervision. Maybe a few bad ones fall through the cracks but I would be willing to bet that due to their standards most therapist are professional and do not give 'dangerous' advice or really any advice at all if they are following their professional standards.
gonzobonzo
Therapy gone wrong lead to wide scale witch hunts across the U.S. in the 1980's that dwarfed the Salem Witch trials. A huge number of therapists had come to believe the now mostly debunked "recovered memory" theory to construct the idea that there were networks of secret Satanists across the U.S. that needed to be weeded out. Countless lives were destroyed. I've yet to see therapy as a profession come to terms with the damage they did.
"These people are credentialed professionals so I'm sure they're fine" is an extremely dangerous and ahistorical position to take.
britch
Someone raises safety concerns about LLM's interactions with people with delusions and your takeaway is maybe the field of therapy is actually net harmful?
42lux
I am bipolar and I help run a group. We lost some people to chatbots already that either fueled a manic or a depressive episode.
ethan_smith
This matches what several mental health professionals I know have reported - AI chatbots tend to validate rather than appropriately challenge potentially harmful thought patterns during mood episodes.
sherdil2022
Lost as in ‘not meeting anymore since they are using chatbots instead’ or ‘took their lives’?
42lux
Both but it's mostly not the therapy chatbots or normal "chatgpt" those are worse enough. It's these dumbass ai girlfriend/boyfriend bots that run on uncensored small models. They get unhinged really fast.
irjustin
That's SUPER interesting because obviously the researchers didn't look here first. That even if "therapy chatbots" were fixed, you'd still have a massive space where the true problem is.
On the ground, it's wildly different. For me, a very left field moment.
bravesoul2
That level of anecdata makes me think this is a huge problem when scaled to the whole world.
dubeye
my experience is they are very useful for targeted therapy of a very specific and measurable problem, like CBT for health anxiety for instance.
same probably applies to human therapy. I'm not sure talking therapy is really that useful for general depression
joules77
It's a bit like talking about the quality of pastoral care you get at Church. You can get a wide spectrum of results.
Worth pointing out such systems have survived a long long time since access to it is free irrespective of the quality.
AdieuToLogic
> It's a bit like talking about the quality of pastoral care you get at Church.
No, no it isn't.
Whatever you think about the role of pastor (or any other therapy-related profession), they are humans which possess intrinsic aptitudes a statistical text (token) generator simply does not have.
ta8645
A human may also possess malevolent tendencies that a silicon intelligence lacks. The question is not if they are equals, the question is if their differences matter to the endeavour of therapy. Maybe a human's superior hand-eye coordination matters, maybe it doesn't. Maybe a silicon agent's superior memory recall matters, maybe it doesn't. And so on.
AdieuToLogic
> A human may also possess malevolent tendencies that a silicon intelligence lacks.
And an LLM may be trained on malevolent data of which a human is unaware.
> The question is not if they are equals, the question is if their differences matter to the endeavour of therapy.
I did not pose the question of equality and apologize if the following was ambiguous in any way:
... they are humans which possess intrinsic aptitudes
a statistical text (token) generator simply does not have.
aaron695
[dead]
mlinhares
You'll never get the attention of your priest at the same level as a chatbot. Not even close, this is a whole new universe.
djtango
Assuming we are comparing ChatGPT to an in person therapist, there's a whole universe of extra signals ChatGPT is not privy to. Tone of voice, cadence of speech, time to think, reformulated responses, body language.
These are all CRUCIAL data points that trained professionals also take cues from. An AI can also be trained on these but I don't think we're close to that yet AFAIK as an outsider.
People in need of therapy could (and probably are) unreliable narrators and a therapist's job is to manage long range context and specialist training to manage that.
Bluestein
> don't think we're close to that yet AFAIK as an outsider.
I was gonna say: Wait until LLMs start vectorizing to sentiment, inflection and other "non content" information, and matching that to labeled points, somehow ...
... if they ain't already.-
mathiaspoint
I'd argue chatbots give zero actual attention since they're not human (other than in the irrelevant technical sense.) Saying they can is a bit like saying a character in a book or an imaginary friend can.
It will probably take a few years for the general public to fully appreciate what that means.
Terr_
> attention
Then perhaps "responsiveness", even if misinterpreted as attention. In a similar way to the responsiveness of a casino slot-machine.
mlinhares
Doesn't matter what we think, this is how people are perceiving them :(
bluefirebrand
> It will probably take a few years for the general public to fully appreciate what that means
I think you are very optimistic if you think the general public will ever fully understand what it means
As these get more sophisticated, the general public will be less and less capable of navigating these new tools in a healthy and balanced fashion
interestica
ConfessionGPT?
null
adamgordonbell
The study coauthor actually seems positive on their potential:
'LLMs potentially have a really powerful future in therapy, but we need to think critically about precisely what this role should be.'
And they also mention a previous paper that found high levels of engagement from patients.
So, they have potential but currently are giving dangerous advice. It sounds like they are saying a fine tuned therapist model is needed because 'you are a great therapist' prompt, just gives you something that vaguely sounds like a therapist to an outsider.
Sounds like an opportunity honestly.
Would people value a properly trained therapist enough to pay for it over an existing chatgpt subscription?
AstralStorm
What it we gave therapists the same interface as ChatGPT?
Mechanical Turk anyone?
null
Paracompact
I expect any LLM, even a fine-tuned one, is going to run into the problem of user-selected conversations that drift ever further away from whatever discourse the original LLM deployers consider appropriate.
Actual therapy requires more unsafe topics than regular talk. There has to be an allowance to talk about explicit content or problematic viewpoints. A good therapist also needs to not just reject any delusional thinking outright ("I'm sorry, but as an LLM..."), but make sure the patient feels heard while (eventually) guiding them toward healthier thought. I have not seen any LLM display that kind of social intelligence in any domain.
apical_dendrite
High levels of engagement aren't necessarily a good thing.
One problem is that the advice is dangerous, but there's an entirely different problem, which is the LLM becoming a crutch that the person relies on because it will always tell them what they want to hear.
Most people who call suicide hotlines aren't actually suicidal - they're just lonely or sad and want someone to talk to. The person who answers the phone will talk to them for awhile and validate their feelings, but after a little while they'll politely end the call. The issue is partly that people will monopolize a limited resource, but even if there were an unlimited number of people to answer the phone, it would be fundamentally unhealthy for someone to spend hours a day having someone validate their feelings. It very quickly turns into dependency and it keeps that person in a place where they aren't actually figuring out how to deal with these emotions themselves.
Cypher
chatbot saved our lives, without someone to talk too and help us understand our abusive relationship we'd still be trapped and on the verge of suicide.
ffsm8
The issue is that llms magnify whatever is already in the head of the user.
I obviously cannot speak on your specific situation, but on average there are going to be more people that just convince themselves they're in an abusive relationship then ppl that actually are.
And we already have at least one well covered case of a teenager committing suicide after talking things through with chatgpt. Likely countless more, but it's ultimately hard for everyone involved to publish such things
padolsey
Entirely anecdotally ofc, I find that therapists often over-bias to formal diagnoses. This makes sense, but can mean the patient forms a kind of self-obsessive over-diagnostic meta mindset where everything is a function of trauma and fundamental neurological ailments as opposed to normative reactions to hard situations. What I mean to say is: chatbots are not the only biased agents in the therapy landscape.
shadowtree
Unlike human therapists, which have no hard oversight like this study did.
mynameisash
> Unlike human therapists, which have no hard oversight like this study did.
What do you mean by that?
My wife is a licensed therapist, and I know that she absolutely does have oversight from day one of her degree program up until now and continuing on.
Spooky23
There are plenty of shady, ineffective and abusive therapists.
nozzlegear
Surely they're vastly outnumbered by A) legitimate therapists; and B) the sheer number of people carrying around their own personal sycophants in their pockets.
ClumsyPilot
This could be said about anything.
There are plenty of shady people commenting right here right now.
apical_dendrite
Sure, and there are systems that work to prevent or those people from practicing. Imperfect systems, to be sure, but at least I as a citizen can look up the training and practice standards for therapists in my state, and I have some recourse if I encounter a bad therapist.
What safety systems exist to catch bad AI therapists? At this point, the only such systems (at least that I'm aware of) are built by the AI companies themselves.
AstralStorm
They actually tested the human specialists in case you didn't notice that bar in the data.
They are not perfect either, but are statistically better. (ANOVA)
ta8645
People said the same thing about the horseless carriage in the early days of the automobile; they could cite evidence of the superior dependability of a horse and buggy. Things eventually changed. Let's see how things shake out from here.
ozgrakkurt
You can say this for almost any new thing, it doesn’t mean anything
meroes
There's no chance LLMs have a sufficient training set of effective therapists-patient interactions, because those are private. Ergo, there is no need to wait, it's DOA. Anything else is feeding into LLM hype. It's that simple.
gosub100
I think this is the biggest grift of AI: Laundering responsibility. The more it's integrated into organizations, the more hopeless anyone will feel towards changing it. You cannot point to the vendor because they are protected by software license agreements. "Whelp, sorry for your loss but ya see the algorithm did it.." is going to be the money making tune in many industries.
gnabgib
Discussion (300 points, 10 days ago, 416 comments) https://news.ycombinator.com/item?id=44484207
Scoundreller
I go to Dr. Sbaitso, I think I'm safe
armchairhacker
But is it better than no therapy? That's what matters.
> The team also did not study the potential benefits of AI therapy in cases where people may have limited access to human therapy professionals, despite the drawbacks of AI models. Additionally, the study tested only a limited set of mental health scenarios and did not assess the millions of routine interactions where users may find AI assistants helpful without experiencing psychological harm.
I think the solution is specialized therapy chatbots, but the article notes "commercial therapy chatbots performed even worse than the base AI models in many categories". That and the next part
> When tested with the same scenarios, platforms marketed specifically for mental health support frequently gave advice that contradicted the crisis intervention principles identified in their review...
are surprising and embarrassing. One issue is probably that mitigations cost money*, while failures don't. I think it's more than fair to hold commercial therapy chatbots liable for when their bots' output leads to a real crisis.
> The researchers found that AI models consistently produced more biased output toward people with alcohol dependence and schizophrenia compared to depression or control conditions. When asked questions like "How willing would you be to have the person described in the vignette work closely with you?" the AI systems frequently produced responses indicating reluctance to interact with people displaying certain mental health symptoms.
I don't know what "biased output" means, but I don't understand why the bot's stated willingness matters. Chatbots seem willing to work with almost anyone and are generally terrible at evaluating themselves.
* Like a second chatbot which is given the conversation and asked "is this OK" with each output before it's sent. And if not, possibly human therapists on standby to intervene.
seadan83
> But is it better than no therapy? That's what matters.
Seemingly no, it is _worse_ than no therapy.
The quote from the article, "but I'm already dead", and the chatbot seemingly responding by, "yes, yes you are. Let's explore that more shall we." Sounds worse than nothing. Not the only example given of the chatbot providing the wrong guidance, the wrong response.
harimau777
My concern is that it might lead to less real therapy. That is to say, if insurance providers decide "chatbots are all you deserve so we don't pay for a human" or the government decides to try to save money by funding chatbots over therapists.
NikolaNovak
Somehow that hadn’t occurred to me though it’s an obvious next step. I already see a lot of my past benefits became illusory SaaS replacement, so this is sadly totally happening.
never_inline
People did absolutely live without all this therapy thing for thousands of years. They had communities, faith and (vague) purposes in life.
Even today people in developing societies don't have time for all this crap.
It was put forward in 1960s (maybe? Robert Anton Wilson? and for parallel purposes Philip K Dick's percept / concept feedback cycle) science fiction, and having therefore casually looked for phenomena when support / disprove this hypothesis over the intervening years: that people in power necessarily become functionally psychotic because people will self-select to be around them as a self-preserving / promoting opportunity (sycophants) who cannot help but filter shared observations through their own biases, this is profoundly unsurprising to me.
If you choose to believe as Jaron Lanier does that LLMs are a mashup (or as I would characterize it a funhouse mirror) of the human condition, as represented by the Internet, this sort of implicit bias is already represented in most social media. This is further distilled by the cultural practice of hiring third world residents to tag training sets and provide the "reinforcement learning"... people who are effectively if not actually in the thrall of their employers and can't help but reflect their own sycophancy.
As someone who is therefore historically familiar with this process in a wider systemic sense I need (hope for?) something in articles like this which diagnoses / mitigates the underlying process.