WebArbiter: A Generative Reasoning Process Reward Model for Web Agents

Yao Zhang, Shijie Tang, Zeyu Li, Zhen Han, Volker Tresp

Keywords: Process Reward Modeling, Multi-step Reasoning, Autonomous Agent, WebAgent, Generative Reasoning

TL;DR: WebArbiter is a reasoning-first, principle-guided reward model that provides interpretable, correctness-aligned supervision for web agents, achieving SOTA performance on the new WebPRMBench and improving real-world trajectory search robustness.

Abstract: Web agents hold great potential for automating complex computer tasks, yet their interactions involve long horizons, multi-step decisions, and actions that can be irreversible. In such settings, outcome-based supervision is sparse and delayed, often rewarding incorrect trajectories and failing to support inference-time scaling. This motivates the use of Process Reward Models (WebPRMs) for web navigation, but existing approaches remain limited: scalar WebPRMs collapse progress into coarse, weakly grounded signals, while checklist-based WebPRMs rely on brittle template matching that fails under layout or semantic changes and often mislabels superficially correct actions as successful, providing little insight or interpretability. To address these challenges, we introduce **WebArbiter**, a reasoning-first, principle-inducing WebPRM that formulates reward modeling as text generation, producing structured justifications that conclude with a preference verdict and identify the action most conducive to task completion under the current context. Training follows a two-stage pipeline: reasoning distillation equips the model with coherent principle-guided reasoning, and reinforcement learning corrects teacher biases by directly aligning verdicts with correctness, enabling stronger generalization. To support systematic evaluation, we release WebPRMBench, a comprehensive benchmark spanning four diverse web environments with rich tasks and high-quality preference annotations. On WebPRMBench, WebArbiter-7B outperforms the strongest baseline, Gemini Flash, by **10.9%**. In reward-guided trajectory search on WebArena-Lite, it surpasses the best prior WebPRM by up to **7.2%**, underscoring its robustness and practical value in real-world complex web tasks.

Submission Number: 4