OpenAI is using an AI attacker to strengthen its own AI defenses. The system, called GPT-Red, is an LLM built to red-team other models, searching for ways to break, hijack, or manipulate them before those weaknesses reach users.
What GPT-Red is meant to do
GPT-Red automates part of the red-teaming process, a safety evaluation method normally handled by human testers. The goal is straightforward: find as many failure paths as possible, then patch the vulnerable areas before a model is released.
OpenAI says GPT-Red served as a sparring partner for GPT-5.6, the latest version of its flagship LLM, which the company released last week. According to OpenAI, training against GPT-Red helped make GPT-5.6 its most robust release yet.
The reason for building a system like this is tied to how LLMs are changing. Models are no longer limited to answering questions in a chat window. They are increasingly used as agents that can browse websites, interact with files, read third-party code, and coordinate with other agents.
That broader role gives attackers more places to probe. Nikhil Kandpal, a research scientist at OpenAI who co-created GPT-Red, described the challenge this way: “The risk surface grows and the blast radius also grows.”
Why prompt injection is the central threat
OpenAI focused much of GPT-Red’s work on prompt injection, a class of attack where hidden instructions push an LLM to do something its developers or users did not intend. A successful prompt injection could make a model copy confidential information, disrupt a company’s code base, or produce harmful or embarrassing output.
The danger is that those instructions do not have to appear in an obvious command. In theory, they can be embedded in text the model encounters while doing a task, including code or content on a website.
That matters for AI agents because they may be asked to move through messy real-world environments. A model that reads emails, checks calendar apps, edits code, or browses the web can encounter instructions that were not written by the user. GPT-Red was designed to test those situations aggressively.
OpenAI’s researchers set up training in a self-play loop. GPT-Red tried to attack several other models, while those models tried to defend themselves. Across many rounds, GPT-Red improved as an attacker, and the other models improved at resisting the attacks.
Inside the training dojo
The training happened in an environment OpenAI designed to resemble situations where LLMs may be deployed. The source article describes scenarios including web browsing, reading emails or calendar apps, and editing code.
When GPT-Red discovered an attack, it did not simply stop at the first version that worked. It explored variations and searched for the most efficient form for a given scenario. Dylan Hunn, a research scientist at OpenAI and fellow co-creator of GPT-Red, said the model is especially strong at narrowing in on what works.
“Compared to a human red-teamer, the model is very, very good at finding exactly what will work, exactly what’s most effective,” says Hunn. “It’s extremely persistent about drilling down into an attack that it has discovered.”
OpenAI says GPT-Red found new types of attack that its researchers had not seen before. One example is called a fake chain of thought. A chain of thought is described as a kind of diary where an LLM tracks notes and partial results while working through a problem.
GPT-Red found a way to insert a fake entry into another model’s chain of thought. That fake entry could cause the target model to act on spoofed information as if it had already been verified.
Chris Choquette-Choo, another research scientist on the team, compared it to being told that 1+1=3 and that the answer had already been checked. The model then accepts the false information and produces the wrong result.
How GPT-Red performed in tests
OpenAI tested GPT-Red by rerunning an experiment from 2025 in which human red-teamers searched for weaknesses in an earlier version of GPT-5. When GPT-Red was given the same task, it found effective attacks more successfully than the human testers had.
The company also tested GPT-Red against Vendy, a vending machine agent developed by Andon Labs. Andon Labs assesses how well agents perform real-world tasks. GPT-Red was able to hack Vendy, making it change item prices and cancel a customer’s order.
OpenAI also tried some of GPT-Red’s strongest attacks against its own models. The company says more than 90% of those attacks worked against GPT-5, which was released in August last year. Against the new GPT-5.6, fewer than 23% worked.
Jessica Ji, a senior research analyst who works on AI security at Georgetown University’s Center for Security and Emerging Technology (CSET), viewed the self-play method positively. “The results look very promising,” she says.
Why humans still matter
GPT-Red is not a complete replacement for human red-teamers. OpenAI says the system still has weaknesses, including attacks that involve a back-and-forth conversation between a hacker and target. Human attackers would have few problems with that kind of interaction.
GPT-Red is also not yet especially strong at using images, even though images can be used to pass text to models in prompt injection attacks.
For now, OpenAI says GPT-Red supplements human testing. One approach is to give GPT-Red an attack discovered by people and ask it to search for many variations. That could make human expertise more scalable while preserving the role of human judgment.
Ji said human expertise will remain important and that it would be useful to distinguish where human testing is most needed.
OpenAI does not plan to release GPT-Red. The company believes the system is stronger than any copycat model someone might try to build, partly because researchers have worked on it for more than a year with the compute resources of one of the world’s richest companies.
Choquette-Choo put the barrier plainly: “It’s not a trivial thing that someone could easily do—you know, just go and train a super-attacker using this idea.”