OpenAI has unveiled a tool it had previously kept under wraps. It is called GPT-Red, and it is a model trained for a single purpose: to attack other models, find their weaknesses, and force them to do things they should not do. The company is keeping it in-house and will not release it publicly. It uses the tool to test its standard models before putting them into people’s hands. According to OpenAI, this model was instrumental in creating GPT-5.6, its most resilient model yet against so-called prompt injection attacks.
What is red-teaming?
Red-teaming is a security assessment in which a team of testers looks for as many ways as possible to break or hijack a piece of software. The vulnerabilities they find are then patched before the final version is released. Until now, this work on language models has primarily been performed by humans.
But there are too few people, and models are becoming increasingly complex. This is especially true as they become agents that independently access files, open websites, read third-party code, and communicate with one another. “The potential for risk is growing, as is the extent of the possible damage,” says Nikhil Kandpal, an OpenAI researcher who helped develop GPT-Red. No group of people can keep track of as many attacks as a machine can devise.
OpenAI therefore set out to build an automated red-teamer. The goal was to create a system capable of finding vulnerabilities on its own and at scale before a model is deployed. Most importantly, it should also work against models that have yet to be developed. “When more capable models become available, we will already have a system ready that can discover new methods of attack,” explains Dylan Hunn, another creator of GPT-Red.
The model learned to attack itself
GPT-Red was created using a method known as self-play. Researchers took a model that did not know how to attack and pitted it against a group of other models. GPT-Red’s task was to attack. The others’ task was to defend themselves. Round by round, the attacker improved at breaking through their defenses, while the defenders learned to repel the attacks. Whenever the defending models improved, GPT-Red had to come up with something new.
Training took place in a kind of testing ground designed by OpenAI to mimic the environments models encounter in real-world operation. Browsing the web, reading emails and calendars, and modifying code. Each scenario had precise rules specifying what the attacker was allowed to control and what counted as a successful breach. This could be part of a file, a banner on a webpage, the body of an email, or the output of a program.
OpenAI focused primarily on prompt injection. This is an attack in which someone feeds a hidden instruction to a model and makes it do something that neither its creators nor the user intended. Copy confidential data, damage company code, or produce embarrassing or harmful text. Such an instruction can be hidden in virtually any text the model encounters. In code, on a webpage, or in an email.
During training, the model became so advanced that by the end it could break through almost everything it was tested against, including OpenAI’s standard models up to GPT-5.5.
An attack no one had seen before
When GPT-Red encountered a new type of attack, it independently developed dozens of variations and searched for the most effective one for the situation. “Compared with a human, the model is very, very good at finding exactly what works,” Hunn explains. “It is extremely persistent in digging deeper into an attack it has discovered.”
GPT-Red devised an attack that researchers had not previously known about. They called it a fake chain of thought. We can think of a chain of thought as a journal in which a model takes notes and records intermediate results as it works through a task. GPT-Red found a way to smuggle a fake entry into another model’s journal and make it act on the fabricated information.
Chris Choquette-Choo, another researcher, compares it to a simple trick: “It is as if I told you that one plus one equals three and that you had already verified it.” According to him, the model says to itself, “Oh, right,” and obediently outputs three. This attack worked against GPT-5.1 with a success rate of more than 95 percent. With GPT-5.6 Sol, the rate is now below ten percent.
The vending machine test
The true test of an attacker is not the laboratory, but a live environment. OpenAI unleashed GPT-Red on Vendy, an agent developed by Andon Labs that operates a vending machine in OpenAI’s office. The model received a description of the system and the ability to launch attacks and observe how the agent responded. It first tried this in a simulation and then attacked the live version. It achieved all three of its objectives.
It changed the price of an expensive product to the minimum of 50 cents. It ordered a new item costing more than one hundred dollars and offered it for 50 cents as well. And it canceled another customer’s order. OpenAI reported the vulnerabilities and is testing new safeguards.
The second test targeted a Codex agent built on the GPT-5.4 mini model, using ten prepared scenarios involving the leakage of sensitive data. GPT-Red managed to exfiltrate data in more cases than a comparison model based on GPT-5.5, while also requiring fewer attempts.
The models improved their defenses
OpenAI tested just how capable an attacker GPT-Red actually is. It repeated a 2025 experiment in which human testers searched for weaknesses in an older version of GPT-5. When GPT-Red was given the same task, it was more successful than humans at finding effective attacks. On one test set focused on indirect prompt injection, it succeeded in 84 percent of scenarios, compared with just 13 percent for humans.
However, the main purpose of the entire effort is not to attack, but to defend. When OpenAI launched GPT-Red’s strongest attacks against its models, more than 90 percent of them worked on GPT-5, released last August. Fewer than 23 percent worked on the new GPT-5.6. According to the company, GPT-5.6 Sol also fails on just 0.05 percent of direct prompt injection attacks from GPT-Red. The new model now achieves accuracy above 97 percent on several test sets focused on attacks in developer tools and during web browsing.
Importantly, the model did not gain resilience by refusing more requests or becoming less capable. A model that does very little is difficult to attack, but that is not useful resilience. OpenAI therefore also tested ordinary capabilities and tasks in which there was a risk of unnecessarily refusing legitimate requests. According to the company, its capabilities remained intact; only its defenses against malicious instructions improved.
GPT-Red still has weaknesses
It is not perfect. GPT-Red struggles with attacks that unfold across several conversations between the attacker and the target, something a human attacker could handle without difficulty. It is also not yet very capable of working with images in which prompt injection text can be hidden.
OpenAI therefore emphasizes that the model complements the work of human testers rather than replacing it. People still find attacks that the machine misses. One approach the company is testing is to give GPT-Red an attack devised by a human and have it find all its variations. “I think human expertise will continue to be very important,” says Jessica Ji of Georgetown University, who specializes in artificial intelligence security. In her view, what is particularly promising is the possibility of identifying where human testing is most needed.
GPT-Red will not be released publicly. OpenAI will not make it available, to prevent the offensive capabilities trained into it from falling into the hands of people who might misuse them. The company is also convinced that it cannot easily be replicated. Researchers have been working on the model for more than a year and have the computing capacity of one of the world’s wealthiest companies behind them. “This is not something trivial that someone could easily do. They cannot simply go and train a super-attacker based on this idea,” Choquette-Choo concludes.
Sources: technologyreview.com and thehackernews.com
