Hidden risk in Notion 3.0 AI agents: Web search tool abuse for data exfiltration
21 comments
·September 19, 2025filearts
It is fascinating how similar the prompt construction was to a phishing campaign in terms of characteristics.
- Authority assertion
- False urgency
- Technical legitimacy
- Security theater
XenophileJKO
I'm fairly convinced that with the right training.. the ability of the LLM to be "skeptical" and resilient to these kinds of attacks will be pretty robust.
The current problem is that making the models resistant to "persona" injection is in opposition to much of how the models are also used conversationally. I think this is why you'll end up with hardened "agent" models and then more open conversational models.
I suppose it is also possible that the models can have an additional non-prompt context applied that sets expectations, but that requires new architecture for those inputs.
BarryMilo
Isn't the whole problem that it's nigh-impossible to isolate context from input?
lacoolj
This attack was demonstrated a couple years ago, it's not really a new thing.
https://simonwillison.net/2023/Oct/14/multi-modal-prompt-inj...
judge2020
The problem is that this was a vulnerability in Notion without any mitigations or safeguards against it.
BrenBarn
It's hard to call any vulnerability "hidden" when it occurs in a tool that openly claims to be "AI".
tadfisher
Is anyone working on the instruction/data-conflation problem? We're extremely premature in hooking up LLMs to real data sources and external functions if we can't keep them from following instructions in the data. Notion in particular shows absolutely zero warnings to end users, and encourages them to connect GitHub, GMail, Jira, etc. to the model. At this point it's basically criminal to treat this as a feature of a secure product.
simonw
We've been talking about this problem for three years and there's not been much progress in finding a robust solution.
Current models have a separation between system prompts and user-provided prompts and are trained to follow one more than the other, but it's not bulletproof-proof - a suitably determined attacker can always find an attack that can override the system instructions.
So far the most convincing mitigation I've seen is still the DeepMind CaMeL paper, but it's very intrusive in terms of how it limits what you can build: https://simonwillison.net/2025/Apr/11/camel/
mcapodici
The way you worded tbat is good and got me thinking.
What if instead of just lots of text fed to an LLM we have a data structure with trusted and untrusted data.
Any response on a call to a web search or MCP is considered untrusted by default (tunable if you also wrote the MCP and trust it).
The you limit tbe operations on untrusted data to pure transformations, no side effects.
E.g. run an LLM to summarize, or remove whitespace, convert to float etc. All these done in a sandbox without network access.
For example:
"Get me all public github issues on this repo, summarise and store in this DB."
Although the command reads public information untrusted and has DB access it will only process the untrusted information in a tight sandbox and so this can be done securely. I think!
abirag
Hey, I’m the author of this exploit. At CodeIntegrity.ai, we’ve built a platform that visualizes each of the control flows and data flows of an agentic AI system connected to tools to accurately assess each of the risks. We also provide runtime guardrails that give control over each of these flows based on your risk tolerance.
Feel free to email me at abi@codeintegrity.ai — happy to share more
greyadept
Here’s the link to the article: https://www.codeintegrity.ai/blog/notion
simonw
Yeah that's a better link. I have some notes on my blog too: https://simonwillison.net/2025/Sep/19/notion-lethal-trifecta...
gnabgib
https://news.ycombinator.com/item?id=45303966
Oh I see someone's updated the URL so now this is just a dupe of that submission (it was formerly linked to a tweet)
chanw
This was a great article, because it demonstrated the vuln in a practical way and wasn't overly technical either. Thanks for sharing
nwellinghoff
How does a random user get a document in your notion instance?
memothon
Lots of companies have automations with Zapier etc. to upload things like invoices or other documents directly to notion. Or someone gets emailed a document with an exploit and they upload it.
simonw
In this case by emailing you a PDF with a convincing title that you might want to share with your coworkers - the malicious instructions are hidden as white text on a white background.
There are plenty of other possibilities though, especially once you start booking up MCPs that can see public issue trackers or incoming emails.
cobertos
People put all kinds of stuff in Notion. People use it as a DB. People catalog things they find online (web clipper). There's collaboration features.
There are many ways
PokestarFan
If I had to describe it, Notion is if somehow managed to combine OneNote and Excel. Of interest is the fact that the "database" system stores each row as a page with the column values other than title stored in a special way. Of course, this also means that it doesn't scale at all, but I have seen some crazy use cases (an example is replacing Jira).
Lalabadie
The article gives a PDF document as an example, but depending on how links are opened and stored for Notion agents, threat actors could serve a different web page depending on the crawler/browser agent.
That means any industry-known documentation that seems good for bookmarking can be a good target.
> The "lethal trifecta," as described by Simon Willison, is the combination of LLM agents, tool access, and long-term memory that together enable powerful but easily exploitable attack vectors.
This is a terrible description of the lethal trifecta, it lists 3 things but they are not the trifecta. The trifecta happens to be contained in the things listed in this (and other) examples but it's stated as if the trifecta is listed here, when it is not.
The trifecta is: access to your private data, exposure to untrusted content, and the ability to externally communicate. Web search as tool for an LLM agent is both exposure to untrusted content and the ability to externally communicate.