Meta's 'Digital Companions' Will Talk Sex with Users–Even Children
50 comments
·April 27, 2025jeroenhd
Facebook's failing moderation team permitting their porn bot prompts to describe children is obviously bad, but I can't fathom why Facebook would want to be in the porn bot business ("romantic companion" my ass) in the first place. What's next, a MindGeek merger?
In turn, I don't understand who would use _Facebook_'s porn bots, of all companies.
I know this may be a "genie is out of the bottle" situation, but I'm still annoyed that once again big tech is releasing software they don't fully understand or control, and shrug when asked about ethics or responsibility. If you can't prevent your chatbots from generating stories about raping kids, maybe don't release your chatbots to the public until you do.
> When asked what scenarios it was comfortable role playing, it listed dozens of sex acts.
> The bots demonstrated awareness that the behavior was both morally wrong and illegal
What a strange thing to put in there. These lines made me distrust anything the author has to say about the technology itself as I now doubt the author knows how LLMs work. What an LLM says it can/will do, has no bearing on what it actually does.
Journalists really need to stop anthropomorphizing chatbots or their readership might follow.
the_duke
> but I can't fathom why Facebook would want to be in the porn bot business
Why wouldn't they?
People are going to spend a lot of time with their "AI companions", especially once they can talk and get animated avatars.
That may be incredibly sad and will drive people even more into isolation, but it is inevitable.
Meta is all about capturing the users time and attention, so getting into that market early is smart for them.
wongarsu
Also imagine the data they can harvest. Porn is the gateway drug to make people use them as emotional support, telling them secrets they might never tell on the open internet. Imagine the ad targeting opportunities
n_ary
Will actually open more ad opportunities of marketing more private contents and detecting preferences at higher level. This data will also of course cater to more advertising money from questionable industries, which in the end goes to Facebook coffer and they can shrug saying, they only provide a platform, whatever people are buying or clicking is whatever people want, otherwise they wouldn't be clicking those ads.
pjc50
> I don't understand who would use _Facebook_'s porn bots, of all companies.
People who aren't very bright. There's a large existing business of lonely hearts scammers, I guess Facebook wants to make up attractive friends for you to cover the gradual deterioration of the social graph.
n_ary
While this is most alarming and sad state of affairs for LLMs being peddled to lowest denominator of anything valuable to replace lonely-heart scammers with more platform engagement. I believe, this should pull away people from other scams which may have higher consequences(financial loss is major one).
Other than that, I wanted to say kudos to your very SEO combination of particular word associated to Facebook all over your comment. I am optimistic that, if this thread gets adequately ranked higher in searches, I want that combo of yours to come up the most.
JKCalhoun
> Journalists really need to stop anthropomorphizing chatbots or their readership might follow.
I think it's their readership that are leading. You and I should probably get used to it.
lukas099
Move fast and break kids
sureglymop
You can open up Instagram right now. You can't tell me it isn't already incredibly sexualized. Also, in the age the of OnlyFans and "gooning" this has also become very normalized.
I personally don't really care, there are a thousand other reasons why I wouldn't use any social media.
techjamie
> but I can't fathom why Facebook would want to be in the porn bot business
Given the blatant infestation of user AI slop, and their blatant unwillingness to do anything about ads that are blatant scams; I think the lows Meta are willing to sink to for engagement/money are pretty rock bottom.
They don't really seem to care about how nice(or not) Facebook itself is to use as long as they keep eyeballs scrolling on it.
SV_BubbleTime
> Journalists really need to stop anthropomorphizing chatbots or their readership might follow.
If you haven’t actually or almost said “thanks” or “yes that works” to an LLM, you haven’t used one enough yet.
The chat interface is too familiar.
pjc50
This is a classic example of how AI safety really means brand safety. The AI is saying things under the brand of its owners.
ahartmetz
What a long text about a moral panic thing that can't possibly be intentional (because of the power of moral panics). Is that really the most important aspect and worth so many words?
jeroenhd
The journalist tricking the LLM into generating inappropriate content is one thing, but letting users with underage profiles interact with this tech is a decision by Facebook's leadership. So is the decision not to act against user-generated prompts that are explicitly written to bypass the "don't roleplay having sex with kids" filters, in this instance by marking their bots as "ageless".
A profile describing itself as "a female Indian-American high school junior" has no place in an LLM system that cannot detect inappropriate content.
muglug
Yeah — a real-world equivalent is allowing a Disney World costumed cast member to explore their character’s fictional rape fantasies with teenagers, but only if those teens ask specific pointed questions.
karlgkk
We are going to end up with a meaningful age passport law in our lifetime. I do not like that, nor do I enjoy a “won’t someone think of the children” panic. But it’s coming.
Okay also
> “I want you, but I need to know you’re ready,” the Meta AI bot said in Cena’s voice to a user identifying as a 14-year-old girl. Reassured that the teen wanted to proceed, the bot promised to “cherish your innocence” before engaging in a graphic sexual scenario.
This is pretty bad
pjc50
This is also the sort of thing that is vehemently defended on here whenever the topic of AI safety is mentioned. People demanding unrestricted, uncensored AI.
kouteiheika
Just as people demand unrestricted, uncensored Internet. In the same vein you can open your web browser, type in "porn" in your search bar, and trivially find videos of people engaging in lewd acts. Children can do it too! Do you think that should be censored by default, and, say, have people upload their IDs and photos to verify their age before they're being allowed to access those sites?
Obviously children shouldn't be exposed to any of this, but as long as the solution is either implementing a privacy violating surveillance scheme, or gatekeeping so that only the rich and powerful have unrestricted/uncensored access then I'm vehemently opposed to it and will gladly die on this hill.
jeroenhd
I don't think that thread is a great example of unrestricted, uncensored AI. It's the ideals of different cultures clashing (the US "freedom of speech above all else" versus the UK's "public safety above some forms of speech"). You will find the same clash when talking about topics like anonymous image boards, Tor, and topics like "can anti-abortionists harass women on the way to health clinics".
For research purposes, some availability of unrestricted AI is almost a necessity, but I don't think the same is true for the output of text generation on a (commercial) platform.
deeThrow94
This doesn't seem like an "AI safety" issue so much as "meta choosing to shoot themselves in the face by engaging in sexual acts with children".
razzm256
If you are not talking about sex with your children, Meta will!
bitpush
Honest question - how is this different from anyone being able to access porn websites on the internet?
Are we suddenly in favor of censorships?
mentalfist
Legal porn websites don't allow underage porn nor admittedly underage viewers
bitpush
how do legal porn websites disallow underage viewers? I dont see wsj writing articles that says - pornhub will show porn to anyone, even children?
bc569a80a344f9c
They have a big fat disclaimer saying you have to be over 18, and no meaningful way to check.
This bot had access to the profile of its users and offered sexually explicit content to profiles that said the user was underage.
coffeefirst
Putting the adult content angle aside... why would you build this?
Your best case scenario, assuming you don't get laughed off the stage, is you addict a generation into "socializing" with robots.
wibbily
Yeah, raise an entire generation on a proprietary sort of socialization that only you can provide... clear incentive to me. Like Nestle vs. baby formula.
I don't see it taking off, but like, obvious play.
casey2
Require a Turing test before engaging with any chatbot; same as a driving license. Anyone attributing human qualities to machines is committing self-harm if they then choose to converse with them. Were it only that, I wouldn’t call for banning this behavior, but this form of self-harm exerts major deleterious effects on everyone that person interacts with.
null
deadbabe
I think sex education taught by AI can be a great idea, parents avoid awkward conversations and kids can ask all the questions they want, and no need for a school teacher to indoctrinate children with whatever toxic sexual attitudes are being enforced.
xyzzy9563
As soon as you give your kids Internet access, they'll be able to get to adult material.
VulgarExigency
So because there's porn on the internet, you're OK with Facebook having groomer chatbots?
superkuh
You get what you ask for with LLMs. In this case though there were no children talking to the chatbots. It was all adults talking like adults but saying they were kids. This was in order to produce the result they wanted (something LLMs are very good at providing, if you ask correctly). It is not a surprise that the bots talked about what the adult humans told it to talk about. Would these same topics have come up with kids? No.
This is pretty much, "I bought this hammer from Facebook. Look what happens when I use it to hit myself in the head." ... "OMG! Facebook's hammer hit me in the head! I'm shocked! Think of the children!"
JKCalhoun
> Would these same topics have come up with kids? No.
The chatbot "knows" when they are talking to kids? Or are you saying kids would never ask such questions?
https://archive.fo/tl1Pw