Assessing Episodic Memory in LLMs with Sequence Order Recall Tasks

Mathis Pink, Vy A. Vo, Qinyuan Wu, Jianing Mu, Javier S. Turek, Uri Hasson, Kenneth A. Norman, Sebastian Michelmann, Alexander Huth, Mariya Toneva

Keywords: Large Language Models (LLMs), Episodic Memory, Long-Term Memory, Working Memory, Long-context, Memory Evaluation, Memory Benchmark, Memory-augmentation, Retrieval-augmented generation, long-document evaluation, neuroscience, cognitive science, NeuroAI

TL;DR: This paper introduces Sequence Order Recall Tasks (SORT) to evaluate episodic memory in LLMs by testing their ability to recall the order of segments from a previously presented sequence.

Abstract:

Current LLM benchmarks focus on evaluating models’ memory of facts and
semantic relations, primarily assessing semantic aspects of long-term memory.
However, in humans, long-term memory also includes episodic memory, which
links memories to their contexts, such as the time and place they occurred. The
ability to contextualize memories is crucial for many cognitive tasks and everyday
functions. Existing benchmarks have poor coverage of episodic memory. To
address the gap in evaluating memory in LLMs, we define episodic memory for
LLMs and introduce Sequence Order Recall Tasks (SORT), which we adapt from
tasks used in cognitive psychology. SORT requires causal LLMs to recall the
correct order of text segments, and provides a general framework that is both easily
extendable and does not require any additional annotations. We present an initial
evaluation dataset, Book-SORT, comprising 36k pairs of segments extracted from 9
books recently added to the public domain. Based on a human experiment with 155
participants, we show that humans can recall sequence order based on long-term
memory of a book. We find that models can perform the task with high accuracy
when relevant text is given in-context during the SORT evaluation. However, when
presented with the book text only during training, LLMs’ performance on SORT
falls short. By evaluating a new aspect of memory, we believe that SORT will aid
in the emerging development of memory-augmented models.

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 12328