A researcher ran 1,008 SVG tests to find out if AI labs are secretly gaming the 'pelican on a bicycle' benchmark

Zero Evidence of Pelicanmaxxing: What 1,008 SVGs Reveal About AI Benchmark Integrity

Every time a frontier lab ships a new model, Simon Willison asks it the same thing: draw a pelican riding a bicycle, in SVG. The results land in the model's launch thread on Hacker News and climb to the top of the discussion 1.

The benchmark now has its own conspiracy theory. If a strong pelican drawing could shape how millions of people perceive a model launch, wouldn't a lab be tempted to slip a few hundred pelican SVGs into its training data? The community has a term for it: "pelicanmaxxing" 1.

Dylan Castillo of Iwana Labs decided to test it. He generated 1,008 SVGs across seven frontier models (GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, and DeepSeek V4 Pro), covering 48 animal-vehicle combinations with three samples each. He scored every image with an LLM judge, then ran a fixed-effects regression to isolate any benchmark-specific boost 1.

The answer: zero evidence of pelicanmaxxing.

Pelicans rank 6th of 8 animals. Bicycles sit second from last among six vehicles. The pelican-bicycle combination lands at 42nd out of 48 prompts.

Why the methodology matters

The instinct when someone reports "no evidence" is to ask how hard they looked. Castillo's design treats the famous prompt as one cell in a matrix. The 48-combination grid (8 animals by 6 vehicles) means every model also draws pelicans on unicycles, cats on skateboards, whales on boats, and 44 other pairings. The regression absorbs the inherent difficulty of each combination through animal-by-vehicle interaction terms, then measures whether any lab deviates from the average on the benchmark-specific cells 1.

The per-lab pelican effects all land between -0.11 and +0.14 on a 1-to-5 judge scale, with a minimum p-value of 0.25. Not close. The per-lab bicycle effects range from Grok 4.5 at -0.18 (p=0.11) to Gemini 3.5 Flash at +0.27 (p=0.022). Only Gemini clears the conventional significance threshold 1.

But that lone signal is exactly what statistics predicts as noise. Castillo ran 21 comparisons at the 0.05 significance level. Pure chance produces approximately 1.05 false positives at that threshold. One appeared. It does not survive a multiple-comparisons correction 1.

The question behind the question

The pelicanmaxxing debate sounds trivial because it is, on its surface, about a bird on a bicycle. But the reflex it exposes runs through the entire AI industry. Every time an informal benchmark goes viral, the same suspicion follows within hours: are labs training on it? The accusation needs no evidence to gain traction. It requires only the assumption that labs will optimize for anything that shapes perception, and the assumption that nobody is checking.

Castillo's experiment is what checking looks like. To cheat on this benchmark undetected, a lab would need to inject enough pelican-bicycle SVGs to move a specific cell of a 48-prompt grid past a regression that controls for drawing difficulty. The grid catches you. The regression catches you. The 47 control cells catch you.

That is the real finding, and it is not about pelicans. The informal benchmarks the AI community treats as cultural touchstones are amenable to the same statistical scrutiny as formal leaderboards. You can ask a rigorous question with a silly prompt. You need 1,008 samples and a fixed-effects model. What you do not need is a Hacker News thread speculating about training data.

The trust deficit in AI evaluation is real. Models are opaque, leaderboards can be gamed, and labs have incentives that do not always align with full transparency. Those conditions make benchmark-contamination suspicion rational as a default posture. But suspicion without methodology is just noise. Castillo's work shows what it takes to convert that posture into an answer, and the answer across seven of the most powerful models shipping today is that none of them needed to see the test in advance.