Karpathy declares the 'pelican on a bicycle' benchmark dead. AI labs may have already killed it.

Andrej Karpathy has declared the pelican benchmark obsolete. He did it by giving Claude Opus 5 the opening paragraph of The Lord of the Rings, a budget of roughly one million tokens (about $10), and a request for a Three.js rendering of the scene. The model ran for about two hours and produced 5,500 lines of code that procedurally placed and animated 3D assets. Karpathy called the result "janky" but signaled a broader shift: we are, in his words, "starting to leave the territory where you'd test an LLM by" asking it to draw a pelican on a bicycle. 1

That single prompt, created by developer Simon Willison in October 2024, became the closest thing AI has to a standardized test 2. Sixty models have attempted it across roughly two years. Their SVG outputs are tracked on a public timeline, and for a period the quality of the drawing served as a quick visual proxy for a model's general coding capability. The gap between a bad pelican and a good one was easy to see, which is why the test spread.

The contamination test that came up empty

The obvious worry about any benchmark that goes viral is that AI labs will train their models on the test itself. Once a model has memorized "Generate an SVG of a pelican riding a bicycle," its output stops measuring spatial reasoning and starts measuring recall. Developer Dylan Castillo checked whether that had happened. The famous prompt occupied one cell among 48. 3

The results undercut the contamination theory. Pelicans rank 6th of 8 animals in mean quality across all models. Bicycles rank second from last among vehicles. The specific pelican-on-a-bicycle combination sits at number 42 out of 48: only six of 48 animal-vehicle pairs produced worse drawings. If labs were training on the famous prompt, it would sit at or near the top. 3

Castillo then fit a fixed-effects regression across all 1,008 images, controlling for the inherent difficulty of each animal-vehicle combination. The largest positive effect, from GLM-5.2 on the pelican-bicycle cell, registered at p=0.12, well within chance. The single result that cleared conventional significance, Gemini 3.5 Flash on bicycles, is exactly what probability predicts from running 21 tests at a 0.05 threshold: about one false positive, and one appeared. 3

The benchmark was not poisoned. It was outgrown.

What Karpathy's replacement measures that the pelican cannot

The Three.js experiment demands something the SVG prompt structurally cannot: sustained reasoning over spatial coordinates, asset orchestration, and animation logic across thousands of lines of code. A static pelican tests whether a model can produce a coherent 2D image in a single pass. A procedural 3D rendering of a literary scene requires the model to assign coordinates to polygon assets, write code that animates them, and maintain coherence across a long output. 1

The Hacker News discussion around Karpathy's post split along familiar lines. Some commenters called the output a genuine capability signal. Another accused Karpathy of acting as a promotional channel for Anthropic, whose Claude Opus 5 produced the output. 4

What the pelican's arc tells anyone choosing a model

The benchmark's two-year lifespan offers a practical framework for model evaluation:

• The test was not contaminated. Castillo's regression found no evidence any lab trained specifically on the pelican prompt. 3
• It saturated. As frontier models improved broadly, the quality gap between the best and worst pelicans narrowed until the prompt stopped discriminating among them.
• Karpathy's Three.js test addresses saturation by demanding a larger, more complex output. But it introduces a new evaluation problem: a static SVG is either a pelican or it is not. A 5,500-line Three.js scene that its own creator called "janky" resists simple scoring.

For teams evaluating models, the lesson is to rotate benchmarks before they become memes. The replacement will face the same clock, and it starts ticking the moment someone screenshots the result and posts it.