Learning Multimodal Trajectory Representations for Web Agent Planning

Xuan Zhang, Shengbo Cai, Ziyan Jiang, Rui Meng, Zora Zhiruo Wang, Lu Guangcheng, Zhiyong Wu, Yanyi Shang, Dehan Kong

Keywords: Web Agent, Retrieval-Augmented Generation

TL;DR: We propose a multimodal trajectory retrieval framework with new datasets and benchmarks, leveraging VLM and optimized contrastive learning, and validate its effectiveness by boosting web agent performance on Online-Mind2Web.

Abstract: Trajectory data, capturing multimodal human actions and states, are pivotal for building autonomous Web agents and transferring skills across tasks, encoding knowledge by compressing past experience into structured Markov sequences. Yet current methods for trajectory modeling remain fragmented, often relying on task-specific heuristics or textual signals. Progress on multimodal trajectories has been limited by the difficulty of representing visual information within long-step histories that exceed model context windows. Hence, how to effectively learn from multimodal trajectories remains a major and insufficiently addressed challenge amid ever-growing datasets. In this work, we introduce Multimodal Trajectory Representation Learning, bridging the gap between universal retrieval and agent-centric trajectory modeling. We construct the Unified Agent Trajectory Dataset (UATD) from annotated demonstrations and states across diverse real-world scenarios. Building on this, we present GAE-Bench, a benchmark containing a large number of trajectory-based retrieval pairs. Our proposed GAE-Retriever, a multimodal retriever based on vision-language models that uses token selection and GradCache to optimize the contrastive objective. Over multiple web-agent datasets, it surpasses strong baselines on retrieval recall. To demonstrate potential downstream applications, we develop WebRAGent, a retrieval-augmented web agent that integrates GAE-Retriever and supports both DOM- and vision-based observations. WebRAGent proves effective on both textual and visual retrieved knowledge, achieving performance gains of 16-22\% vs. non-retrieval on the Online-Mind2Web benchmark.

Submission Number: 44