Curl: We still have not seen a valid security report done with AI help

There are places that will pay bounties on even very flimsy reports to avoid the press / perception that they aren't responding to researchers. But that's only going to remain as long as a very small number of people are doing this.

It's easy for reputational damage to exceed $1'000, but if 1000 people do this...

$1000 is cheap... The real question is when will companies become wise to this scam?

Most companies make you fill in expense reports for every trivial purchase. It would be cheaper to just let employees take the cash - and most employees are honest enough. However the dishonest employee isn't why they do expense reports (there are other ways to catch dishonest employees). There used to be a scam where someone would just send a bill for "services" and those got paid often enough until companies realized the costs and started making everyone do the expense reports so they could track the little expenses.

The git commit hashes in the diff are interesting: 1a2b3c4..d4e5f6a

I think my wetware pattern-matching brain spots a pattern there.

An real report would have a GDB trace that looks like that, so it isn't hard to create such a trace. Many of us could create a real looking GDB trace just as well by hand - it would be tedious, boring, and pointless but we could.

The vast majority of developers are 10-100x more likely to find a security hole in a random tool than spend time improving their reputation on a bug bounty site that pays < 10% their salary.

That makes it extremely hard to build a reputation system for a site like that. Almost all the accounts are going to be spam, and the highest quality accounts are going to freshly created and take ~ 1 action on the platform.

> I feel like the problem seems to me to be behavior, not a technology issue.

To be honest, this has been a grimly satisfying outcome of the AI slop debacle. For decades, the general stance of tech has been, “there is no such thing as a behavioral/social problem, we can always fix it with smarter technology”, and AI is taking that opinion and drowning it in a bathtub. You can’t fix AI slop with technology because anything you do to detect it will be incorporated into better models until they evade your tests.

We now have no choice but to acknowledge the social element of these problems, although considering what a shitshow all of Silicon Valley’s efforts at social technology have been up to now, I’m not optimistic this acknowledgement will actually lead anywhere good.

I consider myself a left leaning soyboy, but this could be the outcome of too "nice" of a discourse. I won't advocate for toxicity, but I am considering if we bolster the self-image of idiots when we refuse to call them idiots. Because you're right, this is fundamentally a people problem, specifically we need people to filter this themselves.

I don't know where the limit would go.

I guess I'm confused by your position here.

> I feel like the problem seems to me to be behavior, not a technology issue.

Yes, it's a behavior issue, but that doesn't mean it can't be solved or at least minimized by technology, particularly as a technology is what's exacerbating the issue?

> It's fundamentally about trust in people.

Who is lacking trust in who here?

I had Google's in-search "AI" invent a command line switch that would have been very helpful... if it existed. Complete with usage caveats and warnings!

This was like two weeks ago. These things suck.

AI red tape.

I wonder if this is an indication that they didn't really understand what you said to begin with.

Is it possible that what happened was an impedance mismatch between you and the engineer such that they couldn’t grok what you told them but ChatGPT was able to describe it in a manner they could understand? Real-life experts (myself included, though I don’t claim to be an expert in much) sometimes have difficulty explaining domain-specific concepts to other folks; it’s not a flaw in anyone, folks just have different ways of assembling mental models.

I often do this - ask a LLM for an answer when I already have it from an expert. I do it to evaluate the ability of the LLM. Usually not in the presence of said expert tho.

If I had a dollar for every time I told someone how to fix something and they did something else...

Let's just say not listening to someone and then complaining that doing something else didn't work isn't exactly new.

>They view it as a shortcut to problem solve and it isn't

Oh but it is, used wisely.

One: it's a replacement for googling a problem and much faster. Instead of spending half an hour or half a day digging through bug reports, forum posts, and stack overflow for the solution to a problem. LLMs are a lot faster, occasionally correct, and very often at least rather close.

Two: it's a replacement for learning how to do something I don't want to learn how to do. Case Study: I have to create a decent-enough looking static error page for a website. I could do an awful job with my existing knowledge, I could spend half a day relearning and tweaking CSS, elements, etc. etc. or I could ask an LLM to do it and then tweak the results. Five minutes for "good enough" and it really is.

LLMs are not a replacement for real understanding, for digging into a codebase to really get to the core of a problem, or for becoming an expert in something, but in many cases I do not want to, and moreover it is a poor use of my time. Plenty of things are not my core competence or anywhere near the goals I'm trying to achieve. I just need a quick solution for a topic I'm not interested in.

Those people weren't engineers to start with.

You should ask yourself why this organization wants engineering advice from a chatbot more than from you.

I doubt the reason has to do with your qualities as an engineer, which must be basically sound. Otherwise why bother to launder the product of your judgment, as you described here someone doing?

If it's not worth writing, it's not worth reading.

It's the 2025 version of lmgtfy.

It just feels to me like a boss walking into a car mechanic's shop holding some random tool, walking up to a mechanic, and:

"Hey, whatcha doin?"

"Oh hi, yea, this car has a slight misfire on cyl 4, so I was just pulling one of the coilpacks to-"

"Yea alright, that's great. So hey! You _really_ need to use this tool. Trust me, it's gonna make your life so much easier"

"umm... that's a 3d printer. I don't really think-"

"Trust me! It's gonna 10x your work!"

...

I love the tech. It's the evangelists that don't seem to bother researching the tech beyond making an account and asking it to write a couple scripts that bug me. And then they proclaim it can replace a bunch of other stuff they don't/haven't ever bothered to research or understand.

Comfyui workflows, fine-tuning models, keeping up with the latest arxiv papers, patching academic code to work with generative stacks, this stuff is grueling.

Here's an example https://files.meiobit.com/wp-content/uploads/2024/11/22l0nqm...

Being dismissive of AI art is like those people who dismiss electronic music because there's a drum machine.

Doing things well still requires an immense amount of skill and exhaustive amount of effort. It's wildly complicated

I mean… I have a fancy phone camera in my pocket too, but there are photographers who, with the same model of fancy phone camera, do things that awe and move me.

It took a solid hundred years to legitimate photography as an artistic medium, right? To the extent that the controversy still isn’t entirely dead?

Any cool images I ask AI for are going to involve a lot less patience and refinement than some of these things the kids are using AI to turn out…

For that matter, I’ve watched friends try to ask for factual information from LLMs and found myself screaming inwardly at how vague and counterproductive their style of questioning was. They can’t figure out why I get results I find useful while they get back a wall of hedging and waffling.

How can you be so harsh on all the new kids with Senior Prompt Engineer in their job titles?

They have to prove to someone that they're worth their money. /s

It is not about stating a source, the bad thing is treating chatGPT as an authoritative source like it is a subject matter expert.

It's a "source" that cannot be reproduced or actually referenced in any way.

And all the other examples will have a chain of "upstream" references, data and discussion.

I suppose you can use those same phrases to reference things without that, random "summaries" without references or research, "expert opinion" from someone without any experience in that sector, opinion pieces from similarly reputation-less people etc. but I'd say they're equally worthless as references as "According to GPT...", and should be treated similarly.

In general those point to the person's understanding being shallow. So far when someone says "GPT said..." it is a new low in understanding, and there is no more to the article they googled or second stackOverflow answer with a different take on it, it is the end of the conversation.

The complaint isn't about stating the source. The complaint is about asking for advice, then ignoring that advice. If one asks how to do something, get a reply, then reply to that reply 'but Google says', that's just as rude.

Well, it is not, but the three "sources" you mention are not worth much either, much like ChatGPT.

All three of those should be followed by "...and I checked it to see if it was a sufficient solution to X..." or words to that effect.

It depends on if they are just repeating things without understanding, or if they have understanding. My issue is that people that say "I asked gpt" is that they often do not have any understanding themselves.

Copy and pasting from ChatGPT has the same consequences as copying and pasting from StackOverflow, which is to say you're now on the hook supporting code in production that you don't understand.

the first 2 bullet points give you an array of answers/comments helping you cross check (also I'm a freak, and even on SO, I generally click on the posted documentation links).

I hope you dealt with them by firing them.

Manage people?

Except if you read the blog post we helped a very confused maintainer when they had this dropped on them with no explanation on hacker news except "oooh potential scary heap vuln"