Web agents' world models learn to predict, not to choose: a rebuilt training objective lifts task success

World models for web agents have been trained on the wrong objective: supervised to reproduce web pages faithfully, when the process reward models that score candidate actions need predictions that tell the outcomes of two different clicks apart. A paper posted 2 September 2026 by a nine-author team from UC Berkeley, the MIT-IBM Watson AI Lab, Cal Poly San Luis Obispo, and Xero fixes the objective instead of the model size, and end-to-end task success on the WebArena-Lite benchmark climbs from 21.82% to 28.48% under an otherwise identical best-of-5 setup (arXiv 2609.02885) 1.

The pipeline the paper targets works like this: the agent samples several candidate actions, a world model predicts the web state each would produce, and a ranker or process reward model (PRM) picks one. Today those world models are trained by supervised next-state prediction to emit a fixed target format: an HTML or AXTree snapshot, or a textual summary. The authors argue this objective is misaligned with the ranker, because the ranker needs predictions that discriminate across candidates, not faithful pages 1.

Why faithful prediction is the wrong target

The paper's worked example, from the WebArena Reddit environment, shows fidelity failing choice in two distinct ways. WebDreamer, a textual-summary method, describes the outcome of a different action than the one queried. WebWorld, a full structured-state method, emits a long AXTree-style representation that largely repeats page structure the queried action never touched. A prediction can be faithful in form and useless for ranking: the detail it omits, or drowns in unchanged structure, is exactly the part where two candidate actions differ 1.

The fix, called predicted-state matching, inverts the target. Given a current state and a queried action, the model generates a textual next-state representation, and a fixed judge model receives only that representation plus two candidate next states: the true result of the queried action, and a state reached by an alternative action from the same page. The prediction counts as correct only if the judge picks the right one. Because the judge never sees the instruction, the history, or the action itself, the representation has to carry the distinguishing information on its own 1.

The dataset that makes wrong outcomes visible

Supervising that objective needs data that linear trajectories lack, since a normal rollout records only the action actually taken. The team built branching data from Go-Browse trajectories in WebArena environments: different trajectories revisit the same browser states, so the authors merge repeated states into a state-action graph, each node a browser state, each directed edge an executed action plus the next state it really produced. A node with several outgoing edges becomes a decision point where alternatives and their real outcomes coexist. The construction collapses 2,839 trajectories into 7,730 branching decision points and 30,920 pairwise matching examples, which works out to exactly four discrimination pairs per decision point on average (30,920 divided by 7,730), a ratio computed from the paper's counts 1.

Same base model, same data, 33 points apart

The decisive experiment is a data-matched baseline: the same Qwen3-8B base, fine-tuned on the same branching data but supervised to generate a fixed AXTree target, which is the standard objective. Across the paper's three evaluation levels, swapping the objective is what moves every number 1:

• Held-out matching benchmark (Qwen3-32B judge, matching accuracy): supervised AXTree baseline 47.77%, predicted-state matching 80.80%, existing world models WebDreamer-7B at 74.51% and WebWorld-8B at 70.17%. The 33.03-point gap between those two same-data, same-backbone models is the training objective's effect.
• WebPRMBench action ranking, averaged over four suites (pairwise accuracy / best-of-N accuracy): action-only ranker 82.02 / 55.80, ranker plus WebWorld-8B predicted states 85.48 / 67.63, ranker plus predicted-state-matching states 89.36 / 72.70.
• WebArena-Lite end-to-end task success with GPT-4o as the policy: single-action ReAct 13.94%, best-of-5 with GPT-4o ranking its own candidates 21.82%, best-of-5 plus state-matching predictions 28.48%.

Two secondary results sharpen the claim. Matching accuracy holds when the judge changes model family (81.26% under a GPT-4o judge, 79.31% under Llama-3.1-70B), so the learned representations are not tuned to one referee 1. And on WebPRMBench, the state-augmented ranker was trained with plain answer-only preference supervision, yet it edges WebArbiter-7B's published pairwise average (89.36 vs 89.19) and trails it on best-of-N (72.70 vs 74.60), where WebArbiter's method uses principle-guided reasoning trained through reasoning distillation and reinforcement learning 1.

Decomposing the end-to-end gain from the paper's numbers: candidate sampling alone adds 7.88 points over ReAct, and the discriminative world model adds 6.66 more, so roughly 46% of the total 14.54-point climb comes from better predictions rather than from having more candidates to rank 1.

On this evidence, the objective was the bottleneck

For builders debugging agents that predict plausibly but choose badly, the reflex is a bigger world model or more trajectories. Here the same 8-billion-parameter base and the same 7,730 decision points swung discrimination accuracy by 33 points when only the training signal changed: an objective problem before a scale problem. The scope is real: training data and the end-to-end run sit in WebArena-family environments, GPT-4o is the only policy tested, matching accuracy is a language-model-judged proxy, and the branches capture the alternatives Go-Browse happened to explore, not every action a page allows 1. The paper surfaced on SciRate, the arXiv overlay where researchers endorse new papers, within a day of posting 2; its project page is at dhruvpendharkar.github.io/dwm 1.