Unauthorized experiment on r/changemyview involving AI-generated comments

I'm pretty sure we saw LLMs in yesterday's thread about the judge. There were a lot of strange comments (stately worded and weird logic that were very LLM like, not just dumb person like) and it wouldn't be surprising as it's an easy tool to weaponize chaos. I'm sure there were bots supporting many different positions. It even looks like some accounts were posting contradictions opinions

In the back of mind I knew it wasn't so, but I had been holding onto the belief that surely I could discern between human and bot, and that bots weren't a real issue where I spent my time anyway. But no. We're at a point where any anonymous public comment is possibly an impersonation. And eventually that "possibly" will have to replaced with "most likely".

I don't know what the solution is or if there even is one.

Agree, this is already happening in mass, if anything this is great to raise awareness and show what can happen.

The mods seem overly pedantic, but I guess that is usually the case on Reddit. If they think for a second that a bunch of their content isn’t AI generated, they are deeply mistaken

So you agree the research and data collected was useless?

If the mere possibility of AI-generated context invalidates an argument, it suggests the standards for discourse were already more fragile than anyone cared to admit.

Historically, emotional narratives and unverifiable personal stories have always been persuasive tools — whether human-authored or not.

The actual problem isn't that AI can produce them; it's that we (humans) have always been susceptible to them without verifying the core ideas.

In that sense, exposing how easily constructed narratives sway public discussion is not unethical — it's a necessary and overdue audit of the real vulnerabilities in our conversations.

Blaming the tool only avoids the harder truth: we were never debating cleanly to begin with.

> I’m also reminded of the experiment that Facebook ran on its users to try to make them depressed. Modifying the news feed algorithm in a controlled way to figure out if they could make users fall into a depression or not.

Study: Experimental evidence of massive-scale emotional contagion through social networks - https://www.pnas.org/doi/full/10.1073/pnas.1320040111 | https://doi.org/10.1073/pnas.1320040111

Reporting:

https://www.theguardian.com/technology/2014/jun/29/facebook-...

https://www.nytimes.com/2014/06/30/technology/facebook-tinke...

It’s been entirely fictional for its whole history but people used to have to come up with their made up stories themselves.

Personal identity and personal anecdotes have an outsized effect on how convincing an argument is. That's why politicians are always trying to tell personal stories that support their campaigns.

I did that myself on HN earlier today, using the fact that a friend of mine had been stalked to argue for why personal location privacy genuinely does matter.

Making up fake family members to take advantage of that human instinct for personal stories is a massive cheat.

By your criteria you would ignore that entire text because there was no argument only identity.

On what basis are we to judge the arguments? Have you done broad primary sociological and economic research? Have you even read the primary research?

In general forums like this we're all just expressing our opinions based on our personal anecdotes, combined with what we read in tertiary (or further) sources. The identity of the arguer is about as meaningful as anything else.

The best I think we can hope for is "thank you for telling me about your experiences and the values that you get from them. Let us compare and see what kind of livable compromise we can find that makes us both as confortable as is feasible." If we go in expecting an argument that can be won, it can only ever end badly because basically none of us have anywhere near enough information.

The merit of the argument, in this example, depends on the identity of the arguer. It is a form of an "argument from authority".

And yet, when people invent sockpuppets to convince others that being extremely tough on immigration is good actually, it's never a generic white guy, but a first generation legal immigrant persona. Or some kind of invented groups like "X for Trump", where X is the group with very low approval ratings in reality.

It's like the identity actually matters a lot in real world, including lived experience.

The identity and opinion are typically linked in normal people. Acting like the only thing arguments are about are logic is an absurd understanding on society. Unless you're talking about math, identity does matter. Hey, even in math identity matters.

You're confusing, as many have, the difference between hypothesis and implementation.

You're missing the obvious: it is the lying that is unethical. Now we're talking about people choosing to use a novel tool to lie en masse. What you're saying is like chastising the horrified onlookers during a firebombing of a city, calling them merely jealous of how much better an arsonist the bomber plane is than any of them.

> suddenly it's "grotesquely unethical."

Not suddenly - it was just as unethical before. Only the price per post went down.

Do you think people weren’t upset about it before?

This kind of argument is like saying cheating democratized passing an exam.

>suddenly it's "grotesquely unethical."

What? No.

You know brand new accounts are highlighted green, right?

Agreed, and I think this is a good thing. The Internet was already full of shills, sockpuppets, propaganda, etc, but now it's really really cheap for anyone to do this, and now it's finally getting to a place where the average person can understand that what they're reading is most likely fake.

I hope this will lead to people being more critical, less credulous, and more open to debate, but realistically I think we'll just switch to assuming that everything we like the sound of is written by real people, and everything opposing is all AI.

I'm pretty sure Reddit as a company couldn't care less it's a bot or AI posting so long as it gets people to upvote it. People say they don't like it, but they keep posting on Reddit instead of leaving.

>What's to stop any malicious actor from posting these same comments?

Nothing, but that is missing the broader point. AI allows a malicious actor to do this at a scale and quality that multiplies the impact and damage. Your question is akin to "nukes? Who cares, guns can kill people too"

To me, the concern is the relative ease of performing a coordinated ‘attack’ on public perception at scale.

You're right; this study would be equally unethical without AI in the loop. At the same time, the use of AI probably allowed the authors to generate a lot more comments than they would have been able to manually, and allowed them to psychologically distance themselves from the generated content. (Or, to put it another way: if they'd had to write these comments themselves, they might have stopped sooner, either because they got tired, or because they realized just how gross what they were doing was.)

This should have been common practice since well before AI was capable of presenting convincing prose. It also could be seen as a corollary of Paul Graham's point in https://www.paulgraham.com/identity.html . It's also an idea that I was raised to believe was explicitly anti-bigoted, which people nowadays try to tell me is explicitly bigoted (or at least problematic).

No system is foolproof. The purpose is to add enough friction that it’s pretty inconvenient to do.

In the example in OP, these are university researchers who are probably unlikely to go to the measures you mention.

As far as I remember this disclaimer has only been on /b/, but yes, I love the turn of phrase. I think I used it in conversation within the last day or two, even.

Are you serious? With services like https://anti-captcha.com/ the bot free anonymous discourse is over for a long time now.

It's in bad faith when people seriously tell you they don't expect something when they make rules against it.

With LLMs anonymous discourse is just even more broken. When reading comments like this, I am convinced this study was a gift.

LLMs are practically shouting it from the rooftops, what should be a hard but well-known truth for anybody who engages in serious anonymous online discourse: We need new ways for online accountability and authenticity.

Sure, but it is dangerous to expect anything else than what the study makes clear. LLMs make manipulation just cheaper and more scalable. There are so many rumors about state sponsored troll farms that I guess this study was a good wake-up call for anyone who is upset now. It's like acting surprised that somebody can send you a computer virus or that the email is not from an African prince who has to get rid of money.

> My guy

> Like really where did you think an army of netizens willing to die on the altar of Masking came from when they barely existed in the real world? Wake up.

This style of commenting breaks several of the guidelines, including:

Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.

Please don't fulminate. Please don't sneer

Omit internet tropes.

https://news.ycombinator.com/newsguidelines.html

Also, the username is an obscenity, which is not allowed on HN, as it trolls the HN community in every thread where its comments appear.

So, we've banned the account.

If you want use HN as intended and choose an appropriate username, you can email us at hn@ycombinator.com and we can unban you if we believe your intentions are sincere.

I am well-aware of the problem and its manifestations so far, which is one reason why, as I mention, I have been concerned about it for a very long time. It just hasn’t become an existential problem yet, but the tools and capabilities to get it there are fast approaching, and I hope we come up with something to fight it.