Opus 5 writes five times more code than its predecessor and mostly fails the quality bar

Opus 5 Writes Five Times the Code of Its Predecessor and Fails the Quality Bar 76 Percent of the Time

Anthropic launched Claude Opus 5 on July 24, 2026 with top marks on a cluster of agentic coding benchmarks: 43.3% on Frontier-Bench, 30.2% on ARC-AGI-3, leading scores on terminal coding, desktop automation, and business workflow tasks. 1 That same Friday, AI coding startup HumanLayer ran the model through a benchmark called SlopCodeBench and surfaced something the launch charts left out. Opus 5 generated five times as many functions and callables as Opus 4.8 on the same challenges. It passed 24% of the time. 2

For teams deploying AI coding agents in production, that gap between output volume and reliability is the number that matters.

Most coding benchmarks hand a model a complete problem and ask for a solution. SlopCodeBench, built by @GOrlanski's lab at UW Madison and released in March 2026, does the opposite. Requirements arrive in stages called checkpoints. The model writes code for checkpoint 1, receives new requirements for checkpoint 2, and must extend the codebase without breaking what came before. A strict pass requires every test from every prior checkpoint to remain green. Introduce a defect at checkpoint 4 and it carries forward unless the model catches and fixes it later, which HumanLayer noted did not happen in practice. 2

Under those conditions, the results were rough. HumanLayer tested three models (Opus 5, Opus 4.8, and Sonnet 5) across three problems totaling 17 checkpoints. No model finished any challenge with everything passing, including the one rated "easy." Opus 5 scored four strict passes (24%). Opus 4.8 and Sonnet 5 each scored one (6%). 2

Opus 5 did improve on its predecessors. But the gain was modest. The original SlopCodeBench paper recorded a 17% strict pass rate for Opus 4.6 and 11% for GPT-5.4, making Opus 5's 24% a step forward rather than a generational leap. 2 Meanwhile, the volume of code Opus 5 generated dwarfed anything the older models produced: five times the functions and callables of Opus 4.8. Output scaled dramatically. Reliability inched forward.

Three of Opus 5's four strict passes came from the opening checkpoints of the easiest problem. Once the work shifted from writing new code to maintaining and extending an existing codebase, every subsequent solution carried at least one defect. 2 The model excelled at starting from scratch. It struggled when the job became iterative.

The 41-indicator quality suite tells the other half of the story. SlopCodeBench tracks metrics across five categories: size, complexity, duplication, decomposition, and rule violations. All three tested models showed significant increases in code smells over the course of each challenge. 2 More code, more complexity, more surface area for bugs, and a strict pass rate that stayed below 25%.

A front-page Hacker News thread on the results drew a useful counterpoint from developers. Several argued that code slop is partly a harness problem, not purely a model problem: constraining an agent to work along a single seam rather than editing in place, or instructing the model to review for simplifications before returning, can reduce the volume of low-quality output. 3 One commenter noted they guard against excessive function generation explicitly in their project configuration. 3

That argument has merit. It also sharpens the finding. The model's default behavior, without intervention, is to produce a large volume of code that fails to pass most of the time. Reliability requires active steering: constraints, prompts, configuration, human review. The productivity gain is real. The autonomy is not.

The test was small: three problems, 17 checkpoints, one person watching for six hours. 2 The benchmark is new and unsaturated. Any single run carries noise. But the direction is consistent: on tasks that resemble real software development, frontier models are generating substantially more code than their predecessors while passing less than a quarter of the time. The metrics the industry celebrates are tracking volume. The metric engineering teams live with is whether the code works. Those two lines are diverging, and right now, volume is winning.