They Asked an A.I. Chatbot Questions. The Answers Sent Them Spiraling

If a cult is preying on people, I'm against that, whether it's a chatbot that's central to their doctrine or some science fiction story or whatever.

Mainstream religions have misled and killed more people than the Zizians, Scientologists and Hare Krisnhas ever have. It's no more crazy to kill in the name of Roko's Basilisk than it is to crusade in the name of the Abrahamic God, it's just more socially acceptable.

You can't draw a line with ideology within some systemic, coercive framework (government censorship) because the ideologies supported by governments and powerful interests will always be allowed, and the ideologies they oppose will always be suppressed, regardless of how poisonous they may be.

I'm not advocating absolute free speech here, but I do believe the proper layer for criticism and censorship (if it has to be called that) is at the level of societies and platforms, and only government in extreme cases (such as libel, false advertising and violent threats.) But that means pushing against misinformation and apathy is always going to be a messy affair.

> Isn't the real problem that people are trusting ChatGPT?

Yes. They should IMO be required to provide very explicit warnings, preferably on each response, that their responses are nonsense, or at best only correct by accident.

Like, you could say "isn't the real problem that smoking is dangerous?" but we do require the manufacturers to provide clear warnings in that case. Or "isn't the real problem that people are using drain cleaners incorrectly and gassing themselves", but many jurisdictions actually restrict the sale of certain substances to professionals for that reason.

You can't just say "people choose to use inherently dangerous products; it's entirely their own fault"; there's a responsibility on the manufacturer and/or the state to _warn_ people that the products are dangerous.

The onus is on organizations to not sell or offer knowingly harmful services to the public. Any judge or jury will interpret a company directly advising the customer to jump to their death as knowingly harmful. Every engineer at an AI company is one dead person in a wealthy family away from legal liability, maybe criminally.

No, not in the US, not unless the patient actually jumped off the top of the 19 story building.

If the statement was recorded or part of a pattern of behavior across multiple clients, all willing to testify, the therapist might lose their license (which I will concede is a more severe life consequence for the average therapist than being sent to jail for a month or 2).

> there's actually an Animatrix episode where some kid escapes the Matrix by committing suicide.

Kid's Story

> https://matrix.fandom.com/wiki/Kid%27s_Story

I don't think this interpretation fits with the last quoted sentence: "You would not fall."

You're right that an LLM could theoretically detect suicidal ideation - the article even mentions that ChatGPT briefly showed Torres a mental health warning before it "magically deleted." So the capability exists, but the implementation is clearly broken.

The more profound issue is that, the way I see it, LLMs are fundamentally multiplicative forces. They amplify whatever you bring to the conversation.

If you're an experienced programmer with solid fundamentals, an LLM becomes a force multiplier - handling boilerplate while you focus on architecture. But if you lack that foundation and vibe code away, you'll get code that looks right but is likely riddled with subtle or not so subtle bugs you can't even recognize.

And I suspect that the same principle applies to mental health. If you approach an LLM with curiosity but healthy skepticism, it can be a useful thinking partner. But if you approach it while vulnerable, seeking answers to existential questions, it doesn't provide guardrails - it amplifies. It mirrors your energy, validates your fears, and reinforces whatever narrative you're constructing.

The "helpfulness" training worsens this. These models are optimized to be agreeable, to match your vibe, to keep you engaged. When someone asks "Am I trapped in the Matrix?" a truly helpful response would be "That sounds like you're going through something difficult. Have you talked to someone about these feelings?" Instead, ChatGPT goes, "Yes! You're special! You're the chosen one! Here's how to unplug! Any tall buildings nearby, Neo?"

Brother like 90% of the freaks in these threads are heavily invested into the current LLM bubble so of course they have to hand wave anything negative.

In your case, "speed reading" and "skimming" are obviously just two flavors of "I didn't actually read this properly but still feel qualified to diagnose everyone involved."

You'd love to see more research? You must've been reading so fast, you breezed right past the Stanford Study, the MIT Media Lab Research and the analysis by McCoy showing GPT-4o affirmed psychotic delusions 68% of the time.

Or do you mean it in the sense of "we have to study this for at least 20 years and let thousands suffer and die before we can determine whether chatbots affirming people's delusions or coaching them on how to kill themselves is a bad thing"?

Your concerns about the slippery slope from "maybe chatbots shouldn't coach people through suicide" to "forced psychiatric evaluations" or "institutionalization" is genuinely unhinged. And, tellingly, it betrays your view of mental healthcare as fundamentally punitive and authoritarian rather than supportive or therapeutic. Big advocate for mental health, you are.

You realize we already regulate countless things to protect vulnerable people from mental harm, right?

We put warning labels on cigarettes. We have age ratings on media. We don't let casinos exploit problem gamblers. We don't let bartenders serve alcohol to visibly intoxicated people. We don't let therapists tell suicidal patients to jump off buildings. None of these regulations are authoritarian overreach - they're basic safety measures that exist because in a healthy society, vulnerable people deserve protection, not exploitation.

But apparently when it's time to consider holding tech to similar standards, we're supposed to just shrug and say, "well, they were probably crazy anyway."

Your position essentially boils down to: "Vulnerable people exist, therefore companies should be free to exploit them." That's not a mental health advocacy position - it's tech industry bootlicking dressed up as concern trolling.

That's why you should feel ashamed.

Why not have no limitations anywhere and let mentally unwell children purchase loaded firearms?

It's almost as if there's a spectrum of costs and benefits, and it's incumbent upon mature members of society to debate them as more information emerges.

When you say that 45 percent of people have a “mental illness” you’re really just talking about the human condition at that point.

Pathological thinking is nothing even vaguely unusual in humans, it is in fact the default state.

The definition of pathological is also a matter of opinion, because we can’t even define what is normal and healthy lol.

By many definitions, most religions are an example of delusional thinking.

ChatGPT isn’t exactly raising the bar on that one, with its AI generated religions.

It feels telling me that LLMs are able to detect when they’re being asked to generate an image that violates copyright and halt but detecting and stopping suicidal ideation in a conversation, no matter how much the user insists upon it? Can’t be done!