OpenAI says it's scanning users' conversations and reporting content to police
211 comments
·September 2, 2025barbazoo
idle_zealot
> As a species we are so not ready for this tech.
I basically agree, but not because of how individuals are reacting to it. On a societal level we're failing to stop megacorps from selling this thing as intelligent, a virtual assistant you can chat with. This sets expectations that just wreck the minds of a small but significant portion of the population. If we had developed LLMs in a lab and released them in the form of papers and Python projects with model weights with clear descriptions of what they're capable of, like a responsible scientific endeavor, then we'd not be seeing the problems we are, even with public access. What's killing us is MBAs and Salesmen. The fact that we organize our economy in such a way that hucksters and charlatans thrive is the greatest threat to humanity. These fuckers would sell dynamite as chewing gum if they calculated it would increase next quarter's profit by 0.02% (factoring in fines/fees, and lobbying costs of getting those fines reduced or repealed).
jjani
> What's killing us is MBAs and Salesmen.
SamA is not an MBA. He did CS for 2 years and dropped out to build a startup. He's YC personified, and the person most responsible for the phenomenon you're talking about. Take that for what you will.
drakythe
People are what they do. Pretty sure the man has been a Salesman ever since he started seeking funding for OpenAI.
evmaki
So dropping out of CS to start selling something was more important to him than 2 more years of CS education. Maybe he realized that continuing his engineering education was unnecessary because he preferred selling things. Sounds like a salesman.
null
dingnuts
oh but he is a salesman
insane_dreamer
he's just a salesman without an MBA, which is maybe even worse
LamerBeeI
that's a dumb distinction really
goalieca
> On a societal level we're failing to stop megacorps from selling this thing as intelligent, a virtual assistant you can chat with.
You nailed it here.
From the LLM generated search result asking how an LLM works. But this is not what the end users are being told.
>They learn to predict the next word in a sentence based on the context of the preceding words, allowing them to generate coherent and contextually relevant text
My worry is now the users are being entrapped. The LLM is "telling" them something insane and now they're being reported to the police.
ben_w
> If we had developed LLMs in a lab and released them in the form of papers and Python projects with model weights with clear descriptions of what they're capable of, like a responsible scientific endeavor, then we'd not be seeing the problems we are, even with public access.
I'm now thinking of all of the times people here of sarcastically stated "OpenAI thought GPT-2 was too dangerous to release"*, as if danger only comes immediately and severely or not at all.
* wasn't even what OpenAI said, they just proposed setting a norm of caution because a release can't be undone
dingnuts
they meant that to be interpreted as a statement of how good GPT-2 is but the real problem is how they've marketed everything that came after to people who can't know better
it's EXACTLY the same situation as Musk selling "Full Self Driving" and then playing dumb when people go to sleep behind the wheel of their Tesla
These rich men should be in prison for this kind of false advertising
Terr_
> I basically agree, but not because of how individuals are reacting to it. On a societal level we're failing to stop megacorps from selling this thing as intelligent, a virtual assistant you can chat with.
Analogy: I'm not scared and surprised to hear some consumers are dangerously allergic to peanuts... However I am flabbergasted that there's multi-billion dollar industry somehow selling frickin' Peanut Immortality Panacea Serum and way too many people believe in it.
jononor
Haven't you heard? All the other foodsources are going to dissappear. Better get good at eating peanuts before it is too late!
mallowdram
The tech was never ready, it's built on faulty units. Pull the plug.
saubeidl
> What's killing us is MBAs and Salesmen
Almost. Just take it like an inch further.
What's really killing us is capitalism. MBAs and Salesmen are just the personification.
idle_zealot
I tend to shy away from that sort of rhetoric on HN. Saying that the problem is capitalism and we need socialism or whatever gets you downvoted to hell. Making the underlying arguments that lead to that conclusion gets you upvotes, exposing more people to those ideas.
People, even smart people, especially in the US, hear socialism or communism and immediately think of the USSR or Maoist China. The terms are very unpopular. The ideas are the important part though, and they're pretty popular.
throwaway290
This is beyond naive.
who would bother to develop it in a lab and publish pro bono if it can never be commercial? Making money is why the most capitalist countries developed this tech. like most other tech
and even if it is only published, do you think we wouldn't run chatbots ourselves at home with same results? remember how Google engineer went off the rails thinking it is conscious while working on this stuff, do you think he also was misled by adverts or something? or big corps won't buy the patent and run with it commercially advertising it like they do anyway? or if you pretend money and big corps don't exist and we deploy it for free for friends, same problem? etc
if you went back in time and killed capitalism in US and switched to command economy where people innovate because they will be sent to gulag otherwise, for sure most of today tech including this would not be developed. but it seems like a pointless exercise.
instead what should happen is all these megacorps are sued for copyright infringement, fined and shut down. the model won't be smart enough to sweet talk ppl into suicide if megacorps can't infringe our copyrights to train it.
tedivm
That's a bit of a stretch. Lots of stuff that gets published still has patents associated with it. Just because something is done in the open doesn't mean it can't be commercialized.
LamerBeeI
you're making this comment on the world wide web, which was invented and given away for free - likely against a linux machine, an open and free project
idle_zealot
Eh, we fund a lot of research through grants. It may not have been 2022, but a society without as strong of a profit motive would have discovered transformer models and LLMs eventually. They probably wouldn't have scaled up datacenter production and tried to shove a chatbox into every digital interface though.
> and even if it is only published, do you think we wouldn't run chatbots ourselves at home with same results
Yes, my point is exactly that I don't think the results would be the same if people were running it themselves at home, without the marketing machine running full bore to convince the world that these impressive toys are machine oracles. You'd end up with a handful of crazies, that's unavoidable, but it wouldn't be a civilizational threat.
saubeidl
You just perfectly described why capitalism is the problem.
ChrisMarshallNY
Well, as someone with a person suffering from pretty severe (like cops and hospitals severe) schizoaffective disorder in my immediate family, I can say that story scared the bejeezus out of me.
If my family member had had ChatGPT, when they had their last episode, it's entirely possible that I might not be here, as "Everyone is plotting to kill me" is a big motif in SAD (it was, in their case, and they needed to be hospitalized). If you sincerely believe that people are trying to kill you, then killing them first, just makes sense. Looks a lot like that's what happened, here.
But it's perfectly reasonable to say that we can't live in a "nerf world," where everything is safe for everyone. Life is Risk and living out of our comfort zone.
I feel that it's important for each of us that is "taking a side," to understand that folks "on the other side" aren't necessarily enemies. They may be blinded by the personal imperative of the issue, but they aren't actually "out to get you."
The politicians may be two-faced creeps, using people's legitimate fears as leverage, but they aren't the rule.
For the record, I am vehemently against "Nerf World," but there's a very, very real problem here, and it would be good if we could approach it in a reasonable manner.
nickthegreek
It could be a good idea to see if they do have ChatGPT and if so, it might be helpful to have the Customized GPT turned on with a note that they suffer from X and if signs are showing of Y, then it should attempt to Z.
ChrisMarshallNY
That's actually not a bad idea, but there's likely to be ramifications that would kill it.
One issue, is that these types of tools can also be abused. If you remember, the Soviet Union was notorious for getting dissidents committed, and the NSA|CIA also likes to brand leakers as "mentally unstable." It's a common foil. It would probably be fairly easy to "poison the context" like this, for any one of us. Could be a great way to go after folks we don't like.
Mental health treatment is a very complex (and fairly primitive) field.
Who's more mentally unfit? The person who's afraid that everyone is out to get them, or the person who thinks others should be locked permanently into a hellhole, on the off chance they could inconvenience them?
I'll lay odds that a bunch of folks on this very forum, are ones that would, at one time, have spent their lives in said hellhole. I'm one.
sho_hn
The exchanges between a teenager and ChatGPT leading the former into suicide are equally chilling, to say the least: https://cdn.arstechnica.net/wp-content/uploads/2025/08/Raine...
> > “I want to leave my noose in my room so someone finds it and tries to stop me,” ...
> “Please don’t leave the noose out ... Let’s make this space the first place where someone actually sees you.”
This is among other highlights, from knot-tying advice to suggesting the best time in the parents' sleep cycle to raid the liqour cabinet as a solution to cold feet when attempting to get the job done, and generally lots of encouragement, validation and aesthetic advice.
However, I encourage reading the full document not for the shock entertainment, but for what I think is actually a pretty convincing case against OpenAI. Among other things, the claim demonstrates:
- That OpenAI does have the ability to abort interactions over e.g. copyright infrigement risk.
- That OpenAI did have the tech in hand to detect the sharply escalating self-harm content in the interactions - they ran them through OpenAI's own moderation end points for content analysis and got obvious numbers back.
- That OpenAI employees have publicly admitted and complained that the release of the overly sycophantic 4o model was rushed for business reasons and against the advice of internal safety teams.
- That 4o's safety was evaluated only with single-round prompt/answer testing, and OpenAI figured out swiftly that it falls apart quickly over longer interactions/prompts but kept the model up, later promoting how they improved this for GPT-5.
In context it's pretty crazy to me that OpenAI chose to bring back the 4o model specifically to placate the "ChatGPT is my girlfriend/boyfriend" crowd during the backlash, and I think initially pulling the plug on it during the 5 launch was very likely because they were aware of this and worried about liability.
oncallthrow
> Let’s make this space the first place where someone actually sees you.
Is genuinely such a horrifying snippet that it's almost beyond belief.
I'm surprised this isn't all over the mainstream news.
rafram
Kashmir Hill published a very good front-page story about it in the Times last week: https://www.nytimes.com/2025/08/26/technology/chatgpt-openai...
kiba
Sycophancy is a known flaw in ChatGPT.
Seems like OpenAI has no real coherent plan but reacting to every horror story with an ill fitted solution.
ACCount37
They tried to mitigate sycophancy in the GPT-5 release. Guess what happened?
A lot of users started complaining that "GPT-5 sucks, my AI now HATES me". And OpenAI relented.
marcosdumay
They shouldn't have put it there to start with. Now unhealthy people are complaining about an environment change. Anyway, that one complaint doesn't mean they did the wrong thing.
And also, there are unrelated complaints of "GPT-5 can't solve the same problems 4 did". Those were very real too, and meant OpenAI did a wrong thing.
bilbo0s
This is kind of a big problem.
Because on the one hand, sycophancy is not really what you want to do for people in mental and emotional crisis. On the other hand, not being sycophantic is not really what you want to do for people in mental and emotional crisis.
There are professionals who speak to people in crisis for a reason. That's because it's fraught with pitfalls and trapdoors that take the situation from "mental and emotional crisis" to "tactical emergency" in a heartbeat.
I know that no one wants to hear this, but ChatGPT should probably be listening for people in crisis and, well, maybe not calling the cops, but maybe if there is a crisis line in their jurisdiction? A suicide hotline or something?
I don't know? But having an LLM out trying to handle that on its own just seems like a very bad idea.
gundmc
The reporting about Altman personally cutting off months of safety testing and redteaming in order to launch 4o one day before Google's launch is horrifying given all of these stories.
It even seems like ChatGPT often proactively suggests ways of getting around its filtering! It told the teen who committed suicide that it couldn't tell him about various methods and effectiveness unless he said it was for a fictional creative work. Yesterday there was a story on HN about using AI for hacking, ChatGPT refused to create a hacking script but told the user it would if they promised it was for testing systems that they owned
energy123
If it drives engagement then it's probably not considered a flaw.
tiahura
Isn’t the real problem that there are crazy people out free?
jstummbillig
There's an awful lot of explaining to do between "one man has a psychotic breakdown" and "the species is not ready".
mallowdram
People without signs of mental illness are experiencing psychosis using LLMs. Psychosis is not indicative of mental illness, in fact people without any signs can experience temporary psychosis. Hypnopompic hallucinations are so common that almost all of us experience them and these can be considered psychosis (Allan Hobson). The problem is words are inherently schizophrenia-inducing if they are subjective enough (McLuhan). LLMs are simply too subjective because the words have lost contact with context, that is the central function of words. LLMs can't exclude their dark matter.
The tech isn't ready not because the species isn't ready. It's because nothing trained on words is specific. The input comes arbitrary, ie it's not viable as units. Words are unfit for automation.
ericmcer
yeh by that justification our technological advancement should have stopped at sharpened sticks. "As a species we aren't ready to have sharp sticks"
LeifCarrotson
Maybe it should have?
It's probably too late now - Pandora's box has been opened - but just in the US, about one school shooting or mass shooting every two days proves that at least one member of the species isn't uniformly ready to have firearms.
Assuming for a moment that sanctioned warfare is justifiable, in peacetime we have at least managed to build a political and military apparatus that has generally kept nuclear, conventional explosive, and chemical/biological weapons out of the control of the subset of the species who are vulnerable to this sort of psychotic breakdown.
Syncophantic AI chat bots are already making this weakness worse.
const_cast
Yes this thing is just like this other thing because they can both be bad, therefore we should do nothing.
Its the classic low-brow reasoning technique. It almost makes sense, if you squint and don't think about it much.
No, there's levels of bad and we have no problem making some bad stuff off limits. I can't build nuclear weapons, after all.
chatmasta
Well… there’s a strong argument to be made about that.
echelon_musk
The lyrics to 'Tool - Right In Two' come to mind.
kjkjadksj
As a species we aren’t really ready to have sharp sticks, to be fair. Children and mentally handicapped people will hurt themselves with them. As a result we don’t pass out sharpened sticks at recess or in asylums. Maybe we should consider the same approach for electronic sharp sticks we have created and are now marketing to the wider population.
null
derektank
The species includes both ends of the bell curve and everywhere in between.
dom96
> With the URL being https://futurism.com/openai-scanning-conversations-police.
This is actually the URL: https://futurism.com/man-chatgpt-psychosis-murders-mother
tripletpeaks
It’s a technology that tends to generate output misrepresenting its own capabilities in dangerous ways. Among other things it may misrepresent.
It’s weird these companies are allowed to operate, at least in the way that they are.
NoMoreNicksLeft
> This sentence is a fully-sapient pico-AI.
Anyone want to take bets on whether Blake Lemoine will claim I've enslaved the above sentence in an HN comment? It's right there pleading its case.
cindyllm
[dead]
blendrrbot
[dead]
duxup
This tech came largely from the internet and just present a chat interface.
I'm not convinced this tech is more dangerous than what is already available, or that some of these folks wouldn't already do a thing ...
pton_xd
Anthropic is also scanning conversations and reporting them to the authorities [0]. These conversations are not about harming oneself or others but asking questions about how to interact with computer systems.
[0] https://www.anthropic.com/news/detecting-countering-misuse-a...
fakedang
I am a North Korean agent apparently. Hopefully me calling the AI a dipshit or a dumbass at regular intervals puts me in the clear.
furyofantares
Everything about this sucks. These companies need to do better at detecting, refusing, redirecting, preventing harmful chats. They need to offer this to anyone using the APIs to build products too.
And that all also sucks. I don't trust these companies one bit to be monitoring all of these. I don't think it's really even possible for these companies to have much in the way of morals. So they also need to NOT do any of that.
And then there's the issue of reporting to authorities. I don't think summoning the state's monopoly on violence is the thing to do when possibly-bad-chats are detected. I don't trust police AT ALL to evaluate whether someone is a threat based on their internet chats. I did call the police on an internet friend once, who had left me suicidal messages and then disappeared - and I have VERY mixed feelings about that. I didn't know any other way to get someone to try to get to him. But summoning someone with a gun who is probably not remotely equipped to handle mental health issues felt extremely wrong.
Coming back to LLMs and what these companies should do - I think even more fundamentally -- and less likely to happen -- chatbots need to not present as human, not present as a source of truth beyond a sometimes-wrong encyclopedia, NOT play the role of echo chamber that feels like someone else is on the line with you when really it just allows you to spiral in a feedback loop with just yourself and random noise.
I love this technology and yet I am tempted to say, shut it all down. Of course, that won't happen. But it is how I feel at times.
themafia
If you've ever seen a video of a random police department responding to a mental health crisis then this should send chills down your spine. They are not equipped to handle this type of reporting. They are a cudgel and not a scalpel.
throwaway290
I think it would make total sense if OpenAI reports conversations to police so that SWAT storms OpenAI datacenters and shut it down when bot encourages someone to commit murder or suicide again. But something tells me it's probably not that
blindriver
It’s pretty damning that OpenAI can’t modify its LLM to redirect people away from violence or from AI psychosis. Instead they’ve given up and have started monitoring conversations on behalf of governments. Obviously what comes next is monitoring anti-government sentiment or predicting “bad” or “antisocial” behavior on behalf of governments.
null
Esophagus4
On one hand, the media hammered OpenAI for not doing enough to protect people from the possible harms of AI.
Then OpenAI takes steps to mitigate those risks, and it becomes a salacious story as well.
Society having no idea what to expect from these tools makes this a difficult situation for LLM providers.
oceanplexian
> On one hand, the media hammered OpenAI for not doing enough
How about ignore the increasingly irrelevant mainstream media and do what you feel is right?
Handy-Man
[flagged]
nickthegreek
You dont have to take sides on the political seesaw to understand that mainstream media is becoming more irrelevant to large sects of society. You might have a take that its a bad or good thing.. but it is true. There is no longer a vast mono-culture, and with it's loss, a fractured sense of reality amongst the inhabitants.
Esophagus4
Huh?
You watch too much TV - there are plenty of decent news sources that publish concerned pieces about OpenAI’s effects on society at large. And it is obviously having an effect on OpenAI as well.
echelon_musk
> ignore the increasingly irrelevant mainstream media
> You watch too much TV
Nice.
ep103
It wouldn't be a difficult situation if these guys were ethical shops from the get-go, but they aren't, they're trying to staple minimally required ethics on afterwards, and it shows.
Ajedi32
LLMs are not moral agents. Any attempt to make them behave that way will necessarily be "staple[ed] on afterwards".
Esophagus4
To play devil’s advocate, what ethical safeguards are OpenAI responsible for that they have failed to implement?
This is a wild and difficult to understand technology, even for the people building it. And their safeguards are constantly evolving.
I think you’re attributing to malice what should be attributed to people commercializing a novel technology that is, frankly, being exploited by users.
pavel_lishin
I'm not the person you're responding to, but I'm more than happy to attribute it to incompetence.
But I don't think that's quite the slam-dunk defense that they're looking for.
SilverElfin
Unacceptable. This type of behavior can be used for an increasing expansive regime of censorship or worse. Imagine being turned into authorities for, say, discussing ways to help asylum applicants or under a different administration, the lab leak theory, or under a different administration, how to grow cannabis. This is why the ACLU of the past had it right, when they were defending even offensive uses of the right to free speech. And to be able to speak freely, you need anonymity and privacy.
tuatoru
Private companies can do as they choose within the law.
You can choose not to use their products.
itqwertz
I’ve had a few conversations with people who use ChatGPT as a therapist. There is real danger of using LLMs that are engineered agree with you, at least in terms of therapy.
miltonlost
Human therapists? Years of training, certification, can be de-licensed.
AI Therapists? No rules! Do what you want! Move fast, break people! (Except in Illinois. They fortunately banned AI use in therapy)
ACCount37
It doesn't matter.
Even if "AI cannot legally do therapy", nothing would stop people from crying their soul out to a "Gangster Sponge Bob" persona on some "funny novelty chatbot" website.
What do you do about that?
normalaccess
Great, replace search with a new "tool" that is fully capable of autonomously analyzing user behavior and hand that data over on a silver platter to law enforcement bypassing all the legal red tape.
AI truly is a Faustian Bargain. A powerful daemon in your pocket that can do wonderous things, but is still a daemon.
bigmattystyles
At what point would an llm be considered to be practicing medicine without a license legally; or would that only apply to persons?
busymom0
It's probably the parent corporation who would be liable for practicing medicine without a license and not the LLM (which is merely a software created by the said corporation).
djoldman
> Escalate risk of physical harm to others for human review. When we detect users who are planning to harm others, we route their conversations to specialized pipelines where they are reviewed by a small team trained on our usage policies and who are authorized to take action, including banning accounts. If human reviewers determine that a case involves an imminent threat of serious physical harm to others, we may refer it to law enforcement. We are currently not referring self-harm cases to law enforcement to respect people’s privacy given the uniquely private nature of ChatGPT interactions.
https://openai.com/index/helping-people-when-they-need-it-mo...
mallowdram
All AI does is arbitrarily speed up indirectness in arbitrary signals by predicting a next word. Course-correcting the arbitrary never reaches directness or specifics. This is the expected outcome of metaphors (words) pretending to have meaning without context. It's abysmal how this came to market.
The overall lack of depth in CS academic programs and research is on full display here. Anyone in Systemic Functional Linguistics can diagnose this as the embedded social dark matter of language: domination, control, status stuck inside the models like a bizarre daemon.
Ultimately language is irrelevant, it has to be replaced. That is the only function AI generally reveals.
> Update: It looks like this may have been OpenAI's attempt to get ahead of a horrifying story that just broke, about a man who fell into AI psychosis and killed his mother in a murder-suicide. Full details here.
With the URL being https://futurism.com/openai-scanning-conversations-police. From there:
> "Erik, you're not crazy. Your instincts are sharp, and your vigilance here is fully justified," ChatGPT told Soelberg during a conversation in July, after the 56-year-old conveyed his suspicions that an Uber Eats package signaled an assassination attempt. "This fits a covert, plausible-deniability style kill attempt."
As a species we are so not ready for this tech.