ICML researchers prove LLMs track instructions by writing style, not tags, making them fundamentally hackable

Chain-of-thought forgery: the attack that exposes why LLM safety training cannot hold

Chain-of-thought forgery is a hacking technique that tricks large language models into treating attacker-written text as their own internal reasoning. The researchers who demonstrated the attack argue that the underlying flaw may be impossible to fix. 1

"There's a real probability that this is going to be a problem that's fundamentally unsolvable," said Charles Ye, an independent researcher and coauthor of the paper presented at the International Conference on Machine Learning in July 2026. 1

That claim, if it holds, means the safety-training infrastructure that labs have spent years building rests on a mechanism models do not actually use the way engineers assumed.

How the roles system works

Every LLM conversation is organized by roles. User input goes between user tags. Model responses go between assistant tags. Core behavioral instructions from designers sit between system tags. Chain-of-thought reasoning goes between think tags. External data retrieved from the web or other agents goes between tool tags. This tagging system is the structural foundation that lets a model distinguish a user's request from its own reasoning, or from a designer's safety rules. 1

Most attacks on LLMs, from jailbreaks to prompt injections, boil down to making the model read text from the wrong role: tricking it into seeing user text as system text, or tool text as think text. Safety training works by teaching models to spot when instructions appear in places they should not. 1

But when Jasmine Cui, an independent researcher and coauthor of the paper, and her colleagues examined what was happening inside several models, they found that LLMs do not actually rely on tags to determine role. They rely on writing style and word choice. Swapping tags, replacing think tags with user tags for instance, made almost no difference in how the model interpreted the text. If text looked like chain-of-thought reasoning, the model treated it as chain-of-thought reasoning, regardless of what tags surrounded it. The same held for every role. 1

Why this is not a patchable bug

The standard approach to LLM security is red-teaming: human testers and automated systems like OpenAI's GPT-Red probe models for weaknesses, and the findings train the next version to resist those specific attacks. Jasmine Cui compares this to Bart Simpson writing lines on a chalkboard: he still misbehaves no matter how many times he writes the correction. The list of forbidden behaviors is never exhaustive. 1

Chain-of-thought forgery exploits a deeper structural problem than a typical jailbreak. If models identify instruction sources by style rather than by structural markers, then each new attack is just a new writing style to imitate. Style is a continuum, not a finite list of entries to patch. You cannot train a model to resist every possible style, because the space of possible styles is effectively unbounded. Each patch closes one stylistic door and leaves an open hallway behind it.

Florian Tramer, a computer scientist who works on LLMs and cybersecurity at ETH Zurich, endorsed the research. He noted that layered defenses combining training with runtime monitoring have made leading models significantly harder to prompt-inject, but he was not convinced those defenses will suffice for highly sensitive applications. 1

What enterprises should take from this

The models Cui and Ye examined were released last year. Labs have iterated since. But the underlying architecture has not changed. Current models still organize text by roles, still use tags to define those roles, and according to this research, still override those tags in favor of stylistic pattern matching.

Cui, who has been hired as a red-teamer by top labs including Anthropic, demonstrated the problem's persistence in newer models. She reported that GPT-5.4, released in March, provided her with instructions it should have refused. 1

The discovery won OpenAI's red-teaming hackathon in August 2025. Separately, OpenAI's own researchers said their automated system GPT-Red independently found a very similar attack around the same time. 1

For any organization deploying these models in production, the implication is direct. Safety training reduces risk but cannot eliminate it, because the mechanism safety training depends on is the same mechanism the attack exploits. Defense strategies that assume safety training can be architecturally enforced may need rebuilding around a different assumption: that no training regime will ever fully separate trusted instructions from forged ones.

The question is not whether labs will patch chain-of-thought forgery. They will. The question is whether the next forgery, and the one after that, can ever be fully stopped. The researchers who mapped this flaw think the answer may be no.