Self-Guided Hierarchical Exploration for Generalist Foundation Model Web Agents

Qianlan Yang, Xiangjun Wang, Danielle Perszyk, Yu-Xiong Wang

Keywords: Web Agent, Foundation Models, VLM, Reinforcement Learning

TL;DR: We propose SAGE, a self-guided hierarchical exploration framework that trains generalist foundation model web agents without human supervision, achieving state-of-the-art performance

Abstract: Foundation models have recently shown strong potential as web agents, capable of interpreting high-level instructions and interacting with complex web interfaces. However, existing training paradigms for these agents often rely on predefined task datasets and curated demonstrations, limiting their scalability, adaptability, and capacity for self-improvement. In this work, we introduce *Self-guided hierArchical exploration for Generalist wEb agents* (SAGE), a new training framework designed to support autonomous skill acquisition through self-guided hierarchical exploration. Our method introduces a three-tier exploration strategy: a pre-exploration phase to build structural understanding of web environments, a top-level exploration strategy to generate a self-evolving curriculum of tasks from easy to hard, and a low-level exploration mechanism that combines planning-based rollouts with step-wise learning to improve policy efficiency. Together, these components form a scalable, supervision-free framework for web agent training. Experimental results on WebVoyager and WebArena demonstrate that our method significantly outperforms prior approaches, enabling foundation model agents to learn complex web tasks with greater generalization and robustness.

Primary Area: Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)

Flagged For Ethics Review: true

Submission Number: 25657