Episodic Memory from Compression Boundaries in Latent Representation Space

David Oneil Campos Ferreira, Priscila Rocha Maia Freitas Ribeiro, EMANUEL BORGES PASSINATO, Diogo Fernandes Costa Silva, Arlindo Rodrigues Galvão Filho

Keywords: Large Language Models, LLM Agents, Episodic Memory, Sparse Autoencoders, Representational Surprise, Unsupervised Learning, Out-of-Distribution Detection.

TL;DR: Unsupervised episodic memory for LLM agents driven by latent representational surprise and Sparse Autoencoders.

Abstract: Long-term memory in Large Language Model (LLM) agents requires selective persistence: only a subset of interactions should be consolidated beyond the current context window. Existing memory systems rely on heuristic importance rules or similarity-based novelty, which remain external to the model’s internal computation. We propose a geometric principle for memory formation: episodic memory can emerge from compression failure in latent representation space. We approximate the manifold of routine LLM activations using Sparse Autoencoders (SAEs) and define representational surprise as reconstruction error relative to this learned manifold. Deviations from routine structure yield elevated residuals, providing an unsupervised, model-internal signal for memory writing. We demonstrate this principle through ReSuME, a surprise-gated memory mechanism that commits turns to memory only when normalized reconstruction error exceeds a calibrated threshold. In long-horizon multi-turn dialogue settings, representational surprise separates routine, critical, and out-of-distribution states, and achieves a superior performance–memory trade-off compared to heuristic and similarity-based baselines under fixed budgets. Covariance-aware normalization further enables robust cross-domain calibration. These results suggest that episodic memory gating in neural agents can be grounded in intrinsic latent geometry rather than externally engineered rules.

Submission Number: 108