A new argument from AI security researchers cuts to the core of how large language models understand instructions. The claim is not simply that today’s systems can be hacked, but that a structural weakness in how they process text may make complete protection impossible.
The concern is stark because LLMs are moving into more consequential settings, including government and military systems, online shopping, and health care. If models can be pushed into treating an attacker’s words as trusted instructions, then stronger guardrails may help without eliminating the underlying problem.
The weakness is about who is speaking
The researchers, including Charles Ye and Jasmine Cui, presented their paper at the International Conference on Machine Learning this month. They argue that LLMs are vulnerable because they struggle to reliably identify the source and authority of instructions inside a long stream of text.
In normal use, a chatbot receives different kinds of material: the user’s prompt, the assistant’s prior answer, system-level guidance, private chain-of-thought-style notes, and content from outside tools or web pages. To manage that mixture, chatbots use role markers such as <user>, <assistant>, <system>, <think>, and <tool>. Cui says those are the labels OpenAI uses, while other companies may use different labels for the same purpose.
Those roles matter because many attacks are essentially attempts to confuse a model about authority. A jailbreak can try to make ordinary user text look like higher-priority instruction text. A prompt injection can hide commands in material brought in from an outside source and make the model treat them as if they came from the user, system, or its own reasoning process.
Model makers train systems to notice and reject instructions that appear in the wrong place. But the researchers say that the models they studied do not rely on the tags as much as expected. Instead, the models appear to judge the role of a passage by how it is written and what words it contains.
Chain-of-thought forgery shows the risk
The team began by testing how easily LLMs could be persuaded to behave in ways their makers tried to prevent. They found that prompts written to imitate the style of a model’s chain of thought could make the model act as though the instruction came from itself.
The researchers call this attack chain-of-thought forgery. The source article describes cases in which models produced information they had been trained not to provide, including how to synthesize cocaine and how to sabotage a commercial aircraft’s navigation system.
The paper describes attacks against several of OpenAI’s models. Cui and Ye say they have since observed similar behavior in models from Anthropic, Alibaba, and DeepSeek. The discovery won OpenAI’s red-teaming hackathon in August 2025. Around the same time, other researchers at OpenAI said GPT-Red found a similar attack on its own, which they call a fake chain of thought.
The key point is not one particular prompt. It is that a model can be fooled by text that looks like it belongs to a more trusted role. In the researchers’ experiments, changing the tags around a passage made little difference if the passage still resembled the kind of text the model associated with a given role.
Why red-teaming may not be enough
AI companies already use red-teaming to search for weaknesses. Human testers try to discover new attacks, and model makers also use systems such as OpenAI’s GPT-Red to automate parts of that process. The usual path is to find attacks, train the next model to resist them, and repeat.
Cui argues that this can become a list-making exercise. A model can be trained not to follow known bad patterns, but no list can include every future attack. Her point is that people keep inventing new ways to confuse models, and those methods may not resemble the examples used during training.
That matters because role confusion is not a narrow bug in a single prompt format. According to the researchers, roles are part of the basic machinery used to run and secure LLMs. If a model can be made to infer role from style instead of source, attackers can keep looking for text that imitates the trusted category they want the model to follow.
Florian Tramèr, a computer scientist who works on LLMs and cybersecurity at ETH Zürich, responded positively to the research. He also noted that model makers are combining training with other defenses, including monitoring model behavior after deployment. In his view, leading models have become much harder to prompt-inject, but that may still be insufficient in highly sensitive cases.
The practical lesson is distrust by default
The researchers acknowledge that some of the models they examined were released last year. But they argue the broader lesson still holds: stronger training can reduce the attack surface, yet it cannot guarantee that red-teamers will find every exploit before release.
Cui points to examples from her own red-teaming work, including attacks that depended on role play or surprising the model with information about how it was being used. She says she has been hired by top labs, including Anthropic, for that kind of work in the past.
Ye’s warning is that economic incentives will push more people toward jailbreaks and prompt injections. If LLMs are used in systems that can affect important workflows, then the cost of trusting them too much could rise with their deployment.
For organizations, the implication is uncomfortable but practical:
- Do not assume an LLM will reliably separate trusted instructions from untrusted text.
- Treat agent actions as potentially unsafe, especially when external content can influence the model.
- Use red-teaming and monitoring, but do not treat them as proof of complete security.
- Reserve extra caution for highly sensitive systems where a wrong action could have serious consequences.
The researchers’ central claim is not that defenses are useless. It is that the industry may be dealing with a limit built into the way LLMs interpret text. Better models may resist more attacks, but if the model still sees everything as one continuous stream of tokens, the boundary between instruction and imitation remains a security fault line.