WebArena Verified: Reliable Evaluation for Web Agents

Amine El hattami, Megh Thakkar, Nicolas Chapados, Christopher Pal

Keywords: webarena, web agent

Abstract: Autonomous web agents increasingly operate in multi-step browser workflows, yet widely used benchmarks can misestimate performance due to underspecified goals and brittle checkers—challenges characteristic of normal benchmark maturation rather than flaws in the paradigm. We present WebArena Verified, a reproducible re-evaluation of WebArena that preserves its containerized environments while strengthening measurement. We audit all 812 tasks, repair misaligned evaluations and clarify ambiguous instructions; replace substring matching with type- and normalization-aware comparators; verify backend state for state-changing tasks; and adopt a structured JSON schema with explicit status codes for deterministic scoring. We provide improved results reporting with template-level macro averages, 95\% confidence intervals, and failure-mode breakdowns. We also introduce WebArena Verified Hard, a 137-task subset that retains difficult cases while reducing evaluation cost by 83\%. On the baseline agent we evaluated, it reduces false negatives by approximately 11\%. WebArena Verified remains drop-in compatible with minimal change to existing agents, supporting faithful and comparable progress. We release our code, data, and evaluation tools in our public repository.

Archival Option: The authors of this submission do *not* want it to appear in the archival proceedings.

Submission Number: 98