Human study on AI spear phishing campaigns

> i would expect better from Meta

I'm surprised you would expect better.

Everything I hear about their processes, everything I experience as a user, says their software development is all over the place.

Uploading a video on mobile web? I get the "please wait on this site" banner and no sign of progress, never completes. An image? Sometimes it's fine, sometimes it forgets rotation metadata. Default feed? Recommendations for sports teams I don't follow in countries I don't live in. Adverts? So badly targeted that I end up reporting some of them (horror films) for violent content, while even the normal ones are often for things I couldn't get if I wanted to such as a lawyer specialising in giving up a citizenship I never had. Write a comment? Sometimes the whole message is deleted *while I'm typing* for no apparent reason.

Only reason I've even got an account is the network effect. If the company is forced to make the feed available to others, I won't even need this much.

If they stopped caring about quality of their core product, what hope a billing system's verification emails?

As a related anecdote. On the phone with amazon refund department I was told to wait for a callback. Got a callback from a number that i obviously didn't get to vet. Then on that call they asked for my email address to email me an Amazon sign in page.

I almost just hung-up because you have the rep urging you to do it and I'm trying to vet that every link is what it says it is before I enter any Amazon info. The cookies from my already signed in session did not apply to thia SSO either. Ended up working out in the end.

I could not believe they have the same flow as the scammers do. This is the same company that regularily sends out warning emails about phishing to me. Go figure.

Meanwhile every time I am expecting a package via USPS or DHL, without fail, I get a scam text message about my incoming package. I never get them when I am not expecting a package. This is using a variety of devices, web shops, etc. Somewhere along the way, there is a data stream being sold or leaked.

Looking at what No Fraud does [0], it sounds like Meta has either spun off the first party hardware store from their usual infra, or straight asked a third party to deal with it, and to insulate their main business they split the email domains.

Most companies are already splitting domains for customer and corporate communication, that's a step in the same direction.

While you're right it sounds fishy as hell, it's also mildly common IMO and understadable, especially when e-commerce is not the main business, and could be a reflection of how anti-phishing provisions are pushing companies to be a lot more protective of the email that comes from their main domain.

[0] https://www.nofraud.com/faq/

> But why do they do such things?

In my experience it's because getting a subdomain set up inside large companies is a MAJOR bureaucratic nightmare, whereas registering a new domain is very easy.

I experienced the exact same thing when I bought the Flipper Zero. A "hacker device" and the email communication following the sale being made was straight out of a phishing email campaign book. I don't remember the details, it has been a while, but it was wild how sketchy the emails looked. I hope they have improved the email templates since.

It's always infuriating getting email from Amazon or my bank "here's signs of potential phishing emails/texts" that doesn't include an exhaustive list of every email address and phone number that that organization will try to contact me from. That should be table stakes when it comes to phishing avoidance, and it's something that can only be done by the business, not the customer.

Yes, like you say, there's always the chance that someone hijacked an official domain - that's where other things like a formal communication protocol ("we will never ask for your password", "never share 2FA codes", "2FA codes are separate from challenge-response codes used for tech support") and rules of thumb like "don't click on shortened links" come in. Defense in depth is a must, but the list of official addresses should be the starting point and it isn't.

I noticed Substack recently switched from "click this link to log in" to "we're emailing you a code; enter the code to log in". Wish other companies would follow this approach.

I just received a corporate IT security training link. From an external address and with a cryptic link. After a previous training which asked us not to trust external emails (spoofable) especially not with unknown links.

IT wasn’t amused when I reported it as phishing attempt.

Email and web browsing relies on “deny lists” rather than “allow lists”. So anything goes but you block bad addresses, rather than nothing until you get permissions/trust/credibility. This helped growth of all the networks but means indefinite whack a mole.

I think (but am not sure) that something using trust networks from the ground up would be better in the long term. Consider anything dodgy until it has built trust relationships.

Eg email servers can’t just go for it. You need time to warm up your IP address, use DKIM etc. People can’t just friend you on FB without your acceptance so it’s a lot safer than email, if still not perfect. A few layers of trust would slow bad actors down significantly.

A trust network wouldn’t be binary. Having eg a bunch of spam accounts all trust each other wouldn’t help getting into your social or business network.

Thoughts from experts?

The same is true for operating systems. Why don't they sandbox properly?

We have sandboxing on mobile apps. Why can't we have the same for desktop?

A good start would be ditching HTML in email. Plain text is perfectly suitable for non-marketing emails (and marketing emails are just chaff at this point anyways).

I’ll die on this hill.

Technologically, email aliases have been working wonders for me in personal use. No idea if it could be rolled out effectively for nontechnical users at an organizational scale though, even with automation.

It also does little against compromised mailboxes - heck, a sufficiently advanced spear fish might even have better chances if the user misunderstands the security improvements this would provide.

But I think other than this, there's not much else to fix. Some people are malicious, others get compromised. No fixing that.

I think for the vast majority of people it doesn't depend on not clicking links.

Chrome 0-days are expensive and aren't going to be wasted on the masses. They'll be sold to dodgy middle eastern countries and used to target journalists or whatever.

If you aren't a high value target you can click links. It's fine.

You can have all of this fancy security. But at the end of the day, the weakest and most vulnerable link in the chain is the human.

I blame Microsoft. As a consumer OS its default stance should be no, the user does not intend to grant god permissions to this embedded or external script when they clicked it. Instead the user should have been challenged with a dialog, do you want to install this App and then execute?

What defines a successful spear phishing? Is it just clicking a link?

My process when I see a sketchy email is to hover over the links to see the domain. Phishing links are obvious to anyone who understands how URLs and DNS works.

But working for a typical enterprise, all links are “helpfully” rewritten to some dumbass phishing detection service, so I can no longer do this.

At my current company I got what I assumed was a phishing email, I hovered over the links, saw they were pointing to some dipshit outlook phishing detection domain, and decided “what the hell, may as well click… may as well see if this phishing detection flags it” [0]…

… and it turns out it was not only not legit, but it was an internal phishing test email to see whether I’d “fall for” a phishing link.

Note that the test didn’t check if I’d, say, enter my credentials into a fraudulent website. It considered me to have failed if I merely clicked a link. A link to our internal phishing detection service because of course I’m not trusted to see the actual link itself (because I’d use that to check the DNS name.)

I guess the threat model is that these phishers have a zero-day browser vulnerability (worth millions on auction sites) and that I’d be instantly owned the moment I clicked an outlook phishing service link, so I failed that.

Also note that this was a “spear phishing” email, so it looked like any normal internal company email (in this case to a confluence page) and had my name on it. So given that it looks nearly identical to other corporate emails, and that you can’t actually see the links (they’re all rewritten), the takeaway is that you simply cannot use email to click links, ever, in a modern company with typical infosec standards. Ever ever. Zero exceptions.

- [0] My threat model doesn’t include “malware installed the moment I click a link, on an up to date browser”, because I don’t believe spear phishers have those sort of vulnerabilities available to burn, given the millions of dollars that costs.

"The cost-effective nature of AI makes it highly plausible we're moving towards an agent vs agent future."

Sounds right. I assume we will all have AI agents triaging our emails trying to protect us.

Maybe we will need AI to help us discern what is really true when we search for or consume information as well. The amount and quality of plausible but fake information is only going to increase.

"However, the possibilities of jailbreaks and prompt injections pose a significant challenge to using language models to prevent phishing."

Gives a hint at the arms race between attack and defense.

This story is extra hilarious to me because a threat actor has made a service named karla.pw that lets you harvest info from infostealers and creds dumps.

Is this the same Karla as in Fight Club?

It's all part of the grand enshittification. Trust in anything Internet is gonna erode without any auth, anonymous internet will slowly die.

I hate the InfoSec generated phishing tests.

They all pass DKIM, SPF, etc. Some of them are very convincing. I got dinged for clicking on a convincing one that I was curious about and was 50/50 on it being legit (login from a different IP).

After that, I added an auto delete rule for all the emails that have headers for our phish testing as a service provider.

I think the idea is that it is a numbers game. If you have a way to inexpensively generate a much higher click-through rate than doing it manually, your success rate will go up with a lower investment.

The average SMB company has people that act very differently with their personal email due to they need to protect their checking account that has $400 in it. But have no such measurement or reluctance with work email. They are "clickers". There are also "repeat clickers", "serial clickers", and "frequent clickers". The only thing this study is doing is automating a small part of the profiling and preparation.

There was yet another study titled "Why Employees (Still) Click on Phishing Links" by NIH. (2020) https://pmc.ncbi.nlm.nih.gov/articles/PMC7005690/

Given the pathology, clicking is the visible and obvious symptom.

If you're using ChatGPT directly as opposed to the API, the system prompts could be driving it.

Also, in section 3.6 of the paper, they talk about just switching fishing email, to email helps.

Or said differently, tell it that it's for a marketing email, and it will gladly write personalized outreach

You can host open source llm offline.

There are many uncensored open-weights models available that you can run locally.

They got IRB approval. The authors framed the emails as part of a marketing study involving "targeted marketing emails."

It seems like “the subject clicked a link in an email” is equated to “being phished”, but I’m not certain that is a good definition.

To my understanding, papers on arXiv are NOT reviewed, which is the primary purpose of the platform. The idea, as I see it, is to allow papers to be published more quickly, at no cost, and freely accessible to everyone online.

I don't think this is a bad thing, because even with peer-reviewed papers, there are cases where they can be completely fabricated but still get published. You shouldn't rely on a single paper alone and should do further research if you have doubts about its content.