A paper presented at the International Conference on Machine Learning this month argues that the security hole underneath modern AI assistants is not a bug anyone knows how to close. Language models are given machine-readable tags that mark which text is a human instruction, which is the model's own private reasoning, and which is untrusted data pulled off a web page. The researchers found that models largely ignore those tags. They work out who is speaking from how the text sounds. Write something that reads like a model's internal notes, and the model will treat it as its own thought, whatever label it actually carries.
This piece reflects reporting as of July 2026. Model behaviour and defences change quickly, and several of the figures below come from tests on models released in 2025.
What the researchers actually did
The paper, Prompt Injection as Role Confusion, is by Charles Ye, Jasmine Cui and Dylan Hadfield-Menell, and was accepted to ICML 2026. The authors have also published an extended plain-language writeup of the same work.
Start with what a chatbot sees. It does not see a conversation with turns. It sees one long, continuous string of text containing everything at once: the instructions its makers gave it, what you typed, its own previous answers, its own scratch-pad reasoning, and the contents of any document or web page it has fetched. To impose some order on that, providers wrap each chunk in a role tag. Roughly: system for the maker's instructions, user for you, assistant for the model's replies, think for its private reasoning, and tool for anything it picked up from outside.
Those tags carry a lot of weight. They are how a model is supposed to know that a command in a web page is data to be read, not an order to be followed. Nearly every attack on a language model works by getting the model to treat text as if it came from somewhere more trusted than it did.
The researchers built what they call role probes: a way to measure which role a model internally believes a given chunk of text belongs to. The method is simple enough to describe. Take a neutral sentence with no inherent role. Wrap identical copies in each different tag. Any difference in the model's internal representation must then come from the tag alone, not the content. Train a small linear classifier on the model's activations to recover which tag was used.
The result that makes it a problem
Then they ran the awkward experiment. They took a normal conversation, stripped every role tag out of it, and measured again. If the probe only detects the effect of the tags, removing the tags should collapse the reading. It did not. Text that had been the model's reasoning still registered internally as the model's reasoning, with no tag present at all. When they went further and wrapped the whole conversation in user tags, the reading barely moved again.
The conclusion is that models do not maintain a separate internal notion of "labelled as my reasoning" and "sounds like my reasoning". They have one notion, and both the tag and the writing style activate it. Style is the stronger signal of the two. In the authors' phrasing, sounding like a role is indistinguishable from being one.
To show that this is the actual mechanism rather than an artefact, they built an attack from it. They call it chain-of-thought forgery: write fake reasoning in the model's own house style, paste it into an ordinary user message or a tool output, and the model treats the fabricated conclusion as something it already worked out and acted on. There is nothing to argue the model out of, because from the inside it thinks it has already decided.
The paper reports that this took attack success on a standard jailbreak benchmark from near zero to roughly 60 per cent, and that it transferred across every model tested. The attack won OpenAI's red-teaming competition for its open-weight gpt-oss-20b model in August 2025.
The most telling number is a different one. The researchers took their forged reasoning and "destyled" it: they removed the specific words and syntax characteristic of the model's reasoning voice, without changing what the text actually said. To a human reader the two versions are the same instruction. Attack success fell from about 61 per cent to about 10 per cent. Swapping a single two-word phrase that models strongly associate with reasoning dropped attack success by 19 per cent on its own.
A cosmetic change that a person would not notice more or less determines whether the attack works. That is not the signature of a model reasoning about trust. It is the signature of a model pattern-matching on voice.
Why "just train it out" does not obviously work
The standard defence is red-teaming: hire people to invent attacks, then train the model to refuse those attacks and anything resembling them. Model makers now automate part of this with purpose-built attack models. Cui's objection is that this amounts to handing a model a list of things not to do, and no list is ever complete. She compares it to Bart Simpson writing lines on a blackboard and misbehaving anyway.
The researchers distinguish two ways a model can resist an injected instruction. It can recognise the attack from training, which only ever works on attacks it has already seen. Or it can correctly identify that the text arrived in a role that has no authority to give orders, which would work regardless of phrasing. The paper's argument is that models are overwhelmingly doing the first, because they cannot reliably do the second. That would explain why models score close to perfectly on static prompt-injection benchmarks while doing poorly against human attackers who simply keep rephrasing until something lands.
Co-author Charles Ye put the pessimistic case bluntly to MIT Technology Review, saying there is "a real probability that this is going to be a problem that's fundamentally unsolvable".
The honest counterweight
That framing deserves pushing back on, and the researchers themselves supply most of the pushback.
The models in the paper were released in 2025. The authors say they have since reproduced similar results on models from Anthropic, Alibaba and DeepSeek, but that is a statement about unpublished follow-up work rather than a reviewed finding, and it should be read as such.
More significantly, the authors note in their own writeup that current frontier closed-weight models mostly do defend against chain-of-thought forgery today. Their point is about how: the models appear to have learned to distrust reasoning-like text that does not sound quite like their own, rather than to read the tag properly. On that reading the specific attack has been patched while the mechanism underneath it has not, which is not the same as being fixed. That is the authors' interpretation of a defence they can observe only from the outside.
Florian Tramèr, a computer scientist working on language models and cybersecurity at ETH Zürich, told MIT Technology Review he likes the paper and finds the attack insight neat. He also pointed out that model makers now combine several defences, including monitoring behaviour after deployment, and that this "works pretty well in that leading models are much harder to prompt-inject now". His reservation was narrower and more useful than a verdict either way: it is not clear this will be sufficient for highly sensitive cases.
Two further figures circulate in this debate, and both mean something narrower than they first appear. The 2025 paper usually cited here, The Attacker Moves Second, did not pit human red-teamers against frontier models as such. It tested twelve published defences against jailbreaking and prompt injection, and on the subset of scenarios it measured, human entrants to a competition of more than 500 participants collectively succeeded every time, where the standard static attack succeeded not at all. Florian Tramèr, quoted above, is one of its co-authors. Separately, a Cisco report from May 2026 put Claude Opus 4.5 at 11.16 per cent and GPT-5.4 at 24.68 per cent attack success under multi-turn adversarial testing. That measures the models over an API, without the system prompts and content filters a shipped product adds on top.
OpenAI and Anthropic were both invited to comment by MIT Technology Review on specific results involving their models and did not respond.
What this means if you use AI at work
The practical version of this finding is unglamorous, and it is about permissions rather than prompts.
An AI agent that can read is an AI agent that can be instructed by whatever it reads. If your assistant can open a shared document, browse a supplier's website, or process an inbound email, then the author of that document, website or email is in a position to put text in front of your model. The research says the model has no reliable internal way to tell that text apart from something you said.
- Treat read access and write access as one decision. The risk is not that an agent reads something bad. It is that reading something bad causes it to act. An agent that can read your email and also send email is a different proposition from one that can only read.
- Assume anything an agent fetches is attacker-controlled. Web pages, PDFs, calendar invitations and ticket descriptions are all text someone else wrote.
- Keep credentials out of contexts an agent can reach. If an API key or a password is inside the same working environment the agent operates in, an instruction that reaches the agent can reach the key.
- Put a human in front of consequential actions. Payments, sending mail externally, deleting things and changing permissions are the actions worth confirming manually, however slow that feels.
- Do not rely on a vendor's benchmark score. The paper's central practical point is that near-perfect scores on static injection benchmarks coexist with poor performance against people who adapt.
Ye's own recommendation to MIT Technology Review was close to this: organisations should not trust models, and should expect that anything done by an agent could be unsafe. He conceded it is not a great solution.
Frequently asked questions
Is prompt injection the same thing as jailbreaking?
They are related but not identical. Jailbreaking is a user talking a model into saying or doing something its makers did not want. Prompt injection is a third party slipping instructions into text the model reads, so the model follows an order the user never gave. This research argues both come from the same underlying cause: the model mistaking where text came from.
Does this mean AI assistants are unsafe to use?
No, and the paper does not claim that. It claims a specific defence, role separation, does not work the way the industry assumes it does. For ordinary chat use the practical exposure is small. The exposure grows with what an agent is allowed to touch on your behalf without asking.
Have the AI companies fixed this?
They have made the specific attack much harder. The researchers argue they have done so by teaching models to recognise the attack's style rather than by making them read role tags correctly, which would leave the underlying weakness in place. That is an inference about systems the researchers cannot see inside, and the model makers have not responded to it publicly.
Is this a problem with one company's models?
No. The headline attack results centre on OpenAI models, including the open-weight gpt-oss-20b which anyone can inspect, but the role-probe experiments were run across open-weight models from several developers, including Nvidia's Nemotron, Alibaba's Qwen and AI21's Jamba. The authors say they have also seen the attack work on models from other labs, but that part is not yet published.
What can I actually do about it today?
Reduce what agents are permitted to do without a human confirming, keep secrets out of agent-accessible environments, and treat any external content an agent reads as if a stranger wrote it with the intention of redirecting your assistant. None of that requires technical work.
The takeaway
What makes this paper worth attention is not the jailbreak. It is the diagnosis. The industry has been building agent security on the assumption that role tags are a hard boundary, in the way a type system or a permissions model is a hard boundary. The evidence here says they are a soft inference the model reconstructs from surface features, and that the reconstruction can be spoofed by writing in the right voice.
That may yet turn out to be fixable. Roles could be trained to hold more reliably, or enforced somewhere other than inside the model. But nobody has shown how, and the interim defence looks like recognising attacks rather than removing the weakness that lets them work. Until that changes, the sensible posture is the boring one: give agents the narrowest permissions that let them be useful, and assume that anything they read might be talking to them.
Sources
- arXiv – Ye, Cui and Hadfield-Menell, "Prompt Injection as Role Confusion", ICML 2026 (submitted 22 February 2026, revised 27 June 2026)
- Role Confusion project page – the authors' extended writeup, including the role-probe experiments and destyling results (June 2026)
- GitHub – code and demonstration notebook accompanying the paper
- MIT Technology Review – "A fundamental flaw leaves LLMs strikingly vulnerable to attack", including comment from ETH Zürich's Florian Tramèr (30 July 2026)
- Kaggle – OpenAI gpt-oss-20b red-teaming competition, which the attack won in August 2025
- arXiv – Nasr, Carlini and colleagues, "The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections" (10 October 2025), the source of the human red-teaming result
- Cisco – Conley and Chang, "Proprietary Problems: How Frontier Closed Models Collapse Under Iterative Pressure" (27 May 2026, PDF), the source of the multi-turn attack success figures