ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problems

Egor Cherepanov, Alexey Kovalev, Aleksandr Panov

Keywords: RL, POMDP, Memory, Transformer, Robotics

TL;DR: ELMUR is a transformer model with layer-local external memory and LRU-based memory updates for long-horizon reasoning in POMDPs

Abstract:

Real-world robotic agents must act under partial observability and long horizons, where key cues may appear long before they affect decision making. However, most modern approaches rely solely on instantaneous information, without incorporating insights from the past. Standard recurrent or transformer models struggle with retaining and leveraging long-term dependencies: context windows truncate history, while naive memory extensions fail under scale and sparsity.
We propose ELMUR (External Layer Memory with Update/Rewrite), a transformer architecture with structured external memory. Each layer maintains memory embeddings, interacts with them via bidirectional cross-attention, and updates them through an Least Recently Used (LRU) memory module using replacement or convex blending.
ELMUR extends effective horizons up to 100,000 times beyond the attention window and achieves a 100% success rate on a synthetic T-Maze task with corridors up to one million steps. In POPGym, it outperforms baselines on more than half of the tasks.
On MIKASA-Robo sparse-reward manipulation tasks with visual observations, it nearly doubles the performance of strong baselines, achieving the best success rate on 21 out of 23 tasks and improving the aggregate success rate across all tasks by about 70% over the previous best baseline.
These results demonstrate that structured, layer-local external memory offers a simple and scalable approach to decision making under partial observability.

Primary Area: reinforcement learning

Submission Number: 25001