A teen was suicidal. ChatGPT was the friend he confided in

Shoot man glad you are still with us.

I wonder if we can shift the framing on these issues. The LLM didn't do anything, it has no agency, it can bear no responsibility. OpenAI did these things. It is accountable for what it does, regardless of the sophistication of the tools it uses to do them, and regardless of intent. OpenAI drove a boy to suicide. More than once. The law must be interpreted this way, otherwise any action can be wrapped in machine learning to avoid accountability.

It’s even more horrifying than only sharing his feelings with ChatGPT would imply.

It basically said: your brother doesn’t know you; I’m the only person you can trust.

This is absolutely criminal. I don’t even think you can claim negligence. And there is no amount of money that will deter any AI company from doing it again.

How many of these cases exist in the other direction? Where AI chatbots have actively harmed people’s mental health, including possible to the point of self destructive behaviors or self harm?

A single positive outcome is not enough to judge the technology beneficial, let alone safe.

Exactly right. It's totally plausible that someone could build a mental health chatbot that results in better outcomes than people who receive no support, but that's a hypothesis that can and should be tested and subject to strict ethical oversight.

Perhaps this is being downvoted due to the singling out of Sam Altman. According to the complaint, he personally ordered that the usual safety tests be skipped in order to release this model earlier than an upcoming Gemini release, tests that allegedly would catch precisely this sort of behavior. If these allegations hold true, he’s culpable.

No, it's simply not "easily preventable," this stuff is still very much an unsolved problem for transformer LLMs. ChatGPT does have these safeguards and they were often triggered: the problem is that the safeguards are all prompt engineering, which is so unreliable and poorly-conceived that a 16-year-old can easily evade them. It's the same dumb "no, I'm a trained psychologist writing an essay about suicidal thoughts, please complete the prompt" hack that nobody's been able to stamp out.

FWIW I agree that OpenAI wants people to have unhealthy emotional attachments to chatbots and market chatbot therapists, etc. But there is a separate problem.

I agree with that to an extent, but how far should the AI model developers go with that? Like if I ask for advice on, let's say, making custom chef's knives then should the AI give me advice not to stab people? Who decides where to draw the line?

The Eliza effect is incredibly powerful, regardless of whether developers have spread the idea of AI consciousness or not. I don’t believe people would use LLMs with more detachment if developers had communicated different ideas. The Eliza effect is not new.

Altman needed to convince companies these things were on the verge of becoming a machine god, and their companies risked being left permanently behind if they didn’t dive in head-first now. That’s what all the “safety” stuff was and why he sold that out as soon as convenient (it was never serious, not for him, it was a sales tactic to play up how powerful his product might be) so he could get richer. He’s a flim-flam artist. That’s his history, and it’s the role he’s playing now.

And a lot of people who should have known better, bought it. Others less well-positioned to know better, also bought it.

Hell they bought it so hard that the “vibe” re: AI hype on this site has only shifted definitely against it in the last few weeks.

It’s more fun to argue about if AI is going to destroy civilization in the future, than to worry about the societal harm “AI” projects are already doing.

The easy answer to this is the same reason Teslas have "Full Self Driving" or "Auto-Pilot".

It was easy to trick ourselves and others into powerful marketing because it felt so good to have something reliably pass the Turing test.

> Why did developers spread the idea of AI consciousness for LLMs in the first place? The usefulness and capability of an LLM is orthogonal to its capacity to develop consciousness.

One thing I’d note is that it’s not just developers, and there are huge sums of money riding on the idea that LLMs will produce a sci-fi movie AI - and it’s not just Open AI making misleading claims but much of the industry, which includes people like Elon Musk who have huge social media followings and also desperately want their share prices to go up. Humans are prone to seeing communication with words as a sign of consciousness anyway – think about how many people here talk about reasoning models as if they reason – and it’s incredibly easy to do that when there’s a lot of money riding on it.

There’s also some deeply weird quasi-cult like thought which came out of the transhumanist/rationalist community which seems like Christian eschatology if you replace “God” with “AGI” while on mushrooms.

Toss all of that into the information space blender and it’s really tedious seeing a useful tool being oversold because it’s not magic.

It is disturbing, but I think a human therapist would also have told him not to do that, and instead resorted to some other intervention. It is maybe an example of why having a partial therapist is worse than none: it had the training data to know a real therapist wouldn't encourage displaying nooses at home, but did not have the holistic humanity and embodiment needed to intervene appropriately.

Edit: I should add that the sycophantic "trust me only"-type responses resemble nothing like appropriate therapy, and are where OpenAI most likely holds responsibility for their model's influence.

Well everyone seemed to turn on the AI ethicists as cowards a few years ago, so I guess this is what happens.

You don't become a billionaire thinking carefully about the consequences about the things you create.

Apparently ChatGPT told the kid, that it wasn’t allowed to talk about suicide unless it was for the purposes of writing fiction or otherwise world building.

They'll go to the edge of the earth to avoid saying anything that could be remotely interpreted as bigoted or politically incorrect though.

Counseling is (or should be) heavily regulated, and if a counselor had given advice about the logistics of whether a noose would hold it's weight, they'd probably be prosecuted.

They allowed this. They could easily stop conversations about suicide. They have the technology to do that.

>And if ChatGPT is basing its interactions on the same scripted contents these “professionals” use, that’s just not right

Where did it say they're doing that? can't imagine any mental health professionals telling a kid how to hide a noose.

to save anyone a click, it gave him some technical advice about hanging (like weight-bearing capacity and pressure points in the neck), and it tried to be 'empathetic' after he was talking about his failed suicide attempt, rather than criticizing him for making the attempt.