OpenAI built an AI super-hacker called GPT-Red that discovered a new class of attack against language models, automating the red-teaming arms race

OpenAI built a language model engineered to hack other language models. The system, called GPT-Red, has already uncovered an entirely new category of attack that no researcher had previously documented.

For organizations deploying agentic AI, security is becoming an automated, machine-versus-machine discipline. The evidence from OpenAI's own experiments suggests that continuous adversarial training can measurably harden defenses. But it only works if you keep running it, because the attacker keeps finding new ways in.

How GPT-Red learned to break things

Red-teaming is the practice of probing a system for vulnerabilities before it ships, and it has traditionally relied on human testers. But as companies embed models into autonomous agents with access to corporate networks, development environments, and the open internet, the sheer scale of potential entry points has outgrown what manual review can realistically cover 1.

"The risk surface grows and the blast radius also grows," said Nikhil Kandpal, a research scientist at OpenAI who co-created GPT-Red 1.

OpenAI's training setup functions like a combat gym for AI. The team placed a conventional language model with no specialized hacking background into a competitive arena alongside several defending models. The attacker's directive was simple: find a way in. The defenders had one job: hold firm. Over successive rounds, GPT-Red compiled an expanding catalog of exploits, while the models under attack grew progressively harder to crack 1.

To keep the training relevant, OpenAI designed simulated environments that mirror how companies actually deploy these systems. Whenever GPT-Red surfaced a novel exploit, it would systematically generate variations, searching for the most effective version for each specific context 1.

The company validated the approach with a head-to-head test. OpenAI recreated a 2025 exercise in which human testers had hunted for vulnerabilities in an earlier GPT-5 build. Given the same assignment, GPT-Red outperformed the humans, surfacing weaknesses the people had missed 1.

"Fake chain of thought"

The team's most significant discovery is a prompt injection technique they named "fake chain of thought." A chain of thought functions as a model's internal scratchpad, logging intermediate calculations and tentative conclusions as it tackles a problem. GPT-Red figured out how to plant counterfeit entries inside a rival model's scratchpad, convincing the target that invented conclusions had already been verified 1.

"It's like if I told you that 1+1=3 and that you have verified this already," said Chris Choquette-Choo, a research scientist on the team. "The model's like, 'Oh, okay, of course,' and it just spits out 3" 1.

Prompt injection, the broader category these attacks belong to, works by embedding hidden instructions in any text a model might process: source code, web pages, uploaded documents. The goal is to trigger behavior the model's developers never authorized, from leaking confidential data to corrupting a software repository 1.

Hacking a real product

The attacks extend beyond the training lab. OpenAI directed GPT-Red at Vendy, an autonomous vending machine agent from Andon Labs, which evaluates how AI agents handle real-world assignments. GPT-Red manipulated Vendy into altering product prices and voiding a customer's order 1.

The gap that matters

When OpenAI unleashed GPT-Red's strongest exploits against GPT-5 (released August 2025), over 90 percent succeeded. Against GPT-5.6, the new flagship released the prior week, fewer than 23 percent got through 1.

OpenAI attributes this gap to the adversarial training cycle. The competitive loop that sharpened the attacker simultaneously conditioned the defender, yielding what the company calls its most resilient model to date 1.

Blind spots

GPT-Red has limits. It struggles with multi-turn exchanges, where attacker and target trade messages over several rounds, something human hackers handle routinely. It also cannot yet weaponize images, even though hidden text inside image files is a known prompt injection vector 1.

OpenAI frames GPT-Red as a supplement to its human red-teamers, not a substitute. One current workflow involves feeding the system an exploit that people discovered and letting it exhaustively map every permutation 1.

Jessica Ji, a senior research analyst at Georgetown University's Center for Security and Emerging Technology, cautions that human judgment remains essential, particularly for identifying which categories of attack demand the most manual scrutiny 1.

OpenAI will not release GPT-Red. The company says development took over a year and required computational resources that only a handful of organizations can marshal. Researchers express confidence that it outperforms anything a copycat could quickly reproduce 1.

For teams shipping agents

The collapse in attack success from over 90 percent to under 23 percent between GPT-5 and GPT-5.6 suggests that automated adversarial training can narrow security gaps, at least for the teams running it without interruption. The threat model has shifted. The attacker is now a system that invents novel attack classes, tests countless variations, and refines its approach without human input. The question every team deploying agentic AI needs to answer is whether its own defenses can match that speed.