Agentic Episodic Control

ACL ARR 2026 January Submission8105 Authors

Keywords: Reinforcement Learning, Large Language Models, Episodic Memory

Abstract:

Reinforcement learning (RL) remains fundamentally limited by poor data efficiency and weak generalization. Prior episodic RL methods attempt to alleviate this via external memory modules, yet suffer from two key limitations: a representation bottleneck caused by shallow encoders, and a retrieval dilemma where episodic memory is accessed indiscriminately.
To address these challenges, we propose Agentic Episodic Control (AEC), a novel architecture that integrates large language models (LLMs) into episodic RL.
AEC uses an LLM-based semantic augmenter to generate semantic representations from raw observations, and a critical state recognizer to selectively retrieve valuable experiences.
This transforms memory usage from passive similarity matching into strategic, context-aware recall.
Across five BabyAI-Text environments, AEC achieves 2–6× higher data efficiency than baselines and is the only method to solve complex tasks like UnlockLocal with over 90% success.
It further demonstrates strong cross-task and cross-environment generalization, maintaining performance even under distribution shifts.
AEC shows that combining LLM-derived priors with reinforcement learning yields more sample-efficient and adaptable agents.

Paper Type: Long

Research Area: AI/LLM Agents

Research Area Keywords: LLM agents, agent memory, reinforcement learning in agents

Contribution Types: Approaches to low-resource settings

Languages Studied: English

Submission Number: 8105