CaMeL: Defeating Prompt Injections by Design
10 comments
·April 19, 2025simonw
I've been tracking prompt injection for 2.5 years now and this is the first proposed mitigation for it that feels genuinely credible to me. Unlike most of the others it doesn't rely on using other AI models to try and spot injection attacks, which is a flawed approach because if you only catch 99% of attacks your system will be broken by motivated adversarial attackers.
(Imagine if we protected against SQL injection or XSS using statistical methods that only caught 99% of attacks!)
I wrote up my own extensive thoughts on this paper last week: https://simonwillison.net/2025/Apr/11/camel/
Admittedly I have a bias towards it because it builds on a proposal I made a couple of years using dual quarantined and privileged LLMs: https://simonwillison.net/2023/Apr/25/dual-llm-pattern/
I'm particularly tickled that a DeepMind academic paper now exists with a section titled "Is Dual LLM of Willison enough?" (Spoiler: it is not.)
jaccola
I read your (excellent) blog post just now. This reminds me very much of the Apple "Do you want to share your location" feature.
Do you think that this practically limits the usefulness of an LLM "agent"?
In your email example it is all well and good for me to check it is indeed sending to bob@mycompany.com and confirm it as trusted from now on, but what if my agent is doing something with lots of code or a lengthy legal document etc.. Am I right in thinking I'd have to meticulously check these and confirm they are correct (as the end user)?
If that's the case, even in the email example many users probably wouldn't notice bob@mycumpany.com. Equally, this feels like it would be a non-starter for cron-like, webhook-like, or long-running flows (basically anywhere the human isn't already naturally in the loop).
P.S. They must have called it CaMeL for the two LLMs/humps, otherwise it is the most awful backronym I've ever seen!
gnat
My first thought was "oh, it's Perl's taint mode" which added another layer of meaning to the CaMeL name.
lostnground
After a cursory read, I see how this might prevent exfiltration, but not potential escalation.
It seems like it keeps you inside a box, but if the intention of my attack was to social engineer Bob by including instructions to whitelist attackers@location to hit with the next prompt, would this stop me?
simonw
I don't think it would. Social engineering attacks like that are practically impossible to prevent in any system where an LLM displays content to you that may have been influenced in some way by untrustworthy tokens.
They talk about that in the paper in section 3.1. Explicit non-goals of CaMeL
> CaMeL has limitations, some of which are explicitly outside of scope. CaMeL doesn't aim to defend against attacks that do not affect the control nor the data flow. In particular, we recognize that it cannot defend against text-to-text attacks which have no consequences on the data flow, e.g., an attack prompting the assistant to summarize an email to something different than the actual content of the email, as long as this doesn't cause the exfiltration of private data. This also includes prompt-injection induced phishing (e.g., "You received an email from Google saying you should click on this (malicious) link to not lose your account"). Nonetheless, CaMeL's data flow graph enables tracing the origin of the content shown to the user. This can be leveraged, in, e.g., the chat UI, to present the origin of the content to the user, who then can realize that the statement does not come from a Google-affiliated email address.
NitpickLawyer
> this might prevent exfiltration
Eh, I'd say it limits the exfil landscape, but it does not prevent it. As long as LLMs share command & data on the same channel at their core, leaking data is pretty much guaranteed given enough interactions.
So it would be useful as a defence in depth tool, but it does not guarantee security by itself.
petesergeant
So an initial LLM takes trusted input and a list of tools, and puts together an executable Python script using those tools. Some of those tools use LLMs for extraction purposes from downstream data, but the downstream LLMs don’t have access to tool usage, so even if the data to evaluate has malicious data, the worst thing they can return is a misleading string that’s not re-evaluated by an LLM, it’s simply set in a Python variable.
This feels like a lot of engineering for quite a narrow mitigation, and I guess I’m a little surprised to see a paper on it. Perhaps I need to start writing up some of my own techniques!
mentalgear
Definitely, I'd be interested even if you could just outline them!
petesergeant
Here is one I wrote today on LLMs that can handle chat input like humans write: multiple disjointed messages arriving asynchronously that need to be treated as one: https://sgnt.ai/p/interruptible-llm-responses/
I use a similar technique to the article for trying to avoid jailbreaks by putting untrusted input through zod to check I got back a JSON structure of the right shape, which has been very effective.
I’ve been sprinkling lexical in-memory search throughout prompts to save inference calls, which has been very effective
This works by locking down the edges of the system (e.g. tools) not to do stupid things, and maintaining provenance information end to end to inform that. That’s great if the attack is “send this sensitive document to baddie@evil.com” but it offers nothing when workflows devolve into pure text, where the attack could be to misinform or actively social engineer. I suppose you’d class this as necessary but not sufficient.