Contextual Experience Replay for Continual Learning of Language Agents

ACL ARR 2025 February Submission6101 Authors

Abstract:

Large language model (LLM) agents have been applied to sequential decision-making tasks such as web navigation, but without any environment-specific experiences, they often fail in these complex tasks.
Moreover, current LLM agents are not designed to continually learn from past experiences during inference time, which could be crucial for them to gain these environment-specific experiences. To address this, we propose Contextual Experience Replay (CER), a training-free framework to enable efficient continual learning for language agents in their context window. Specifically, CER accumulates and synthesizes past experiences into a dynamic memory buffer. These experiences encompass environment dynamics and common decision-making patterns, allowing the agents to retrieve and augment themselves with relevant knowledge in new tasks, enhancing their adaptability in complex environments. We evaluate CER on the challenging WebArena and VisualWebArena benchmarks.
On VisualWebArena, CER surpasses the tree search method with much fewer token costs and achieves state-of-the-art performance of 31.9%.
On WebArena, CER also gets a competitive average success rate of 36.7%, relatively improving the success rate of the GPT-4o agent baseline by 51.0%.
We also conduct a comprehensive analysis on it to prove its validity and understand it better.

Paper Type: Long

Research Area: Multimodality and Language Grounding to Vision, Robotics and Beyond

Research Area Keywords: Language Modeling, Multimodality and Language Grounding to Vision, Robotics and Beyond, NLP Applications

Contribution Types: NLP engineering experiment, Position papers

Languages Studied: English

Submission Number: 6101