Assessing Episodic Memory in LLMs with Sequence Order Recall Tasks

ICLR 2025 Conference Submission12328 Authors

Keywords: Large Language Models (LLMs), Episodic Memory, Long-Term Memory, Working Memory, Long-context, Memory Evaluation, Memory Benchmark, Memory-augmentation, Retrieval-augmented generation, long-document evaluation, neuroscience, cognitive science, NeuroAI

TL;DR: This paper introduces Sequence Order Recall Tasks (SORT) to evaluate episodic memory in LLMs by testing their ability to recall the order of segments from a previously presented sequence.

Abstract:

Current LLM benchmarks focus on evaluating models’ memory of facts and
semantic relations, primarily assessing semantic aspects of long-term memory.
However, in humans, long-term memory also includes episodic memory, which
links memories to their contexts, such as the time and place they occurred. The
ability to contextualize memories is crucial for many cognitive tasks and everyday
functions. Existing benchmarks have poor coverage of episodic memory. To
address the gap in evaluating memory in LLMs, we define episodic memory for
LLMs and introduce Sequence Order Recall Tasks (SORT), which we adapt from
tasks used in cognitive psychology. SORT requires causal LLMs to recall the
correct order of text segments, and provides a general framework that is both easily
extendable and does not require any additional annotations. We present an initial
evaluation dataset, Book-SORT, comprising 36k pairs of segments extracted from 9
books recently added to the public domain. Based on a human experiment with 155
participants, we show that humans can recall sequence order based on long-term
memory of a book. We find that models can perform the task with high accuracy
when relevant text is given in-context during the SORT evaluation. However, when
presented with the book text only during training, LLMs’ performance on SORT
falls short. By evaluating a new aspect of memory, we believe that SORT will aid
in the emerging development of memory-augmented models.

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 12328