Understanding the Mechanics and Dynamics of Memorisation in Large Language Models: A Case Study with Random Strings

Till Speicher, Aflah Mohammad Khan, Qinyuan Wu, Vedant Nanda, Soumi Das, Bishwamittra Ghosh, Krishna P. Gummadi, Evimaria Terzi

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: language models, memorization

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We investigate memorization in large language models using random strings and show that tokens are memorized independently and that global and local context both play a distinct role in recollection.

Abstract: Understanding whether and to what extent large language models (LLMs) have memorised training data has important implications for the privacy of its training data and the reliability of its generated output. In this work, we focus on the more foundational question of how LLMs memorise training data. To this end, we systematically train LLMs of different sizes to memorise random token strings of different lengths and different entropies (i.e., sampled from different alphabet distributions) and study their ability to recall the strings. We observe many striking memorisation dynamics including (i) memorisation in phases with the alphabet distributions in the random strings being learnt before their relative positions in the string are memorised and (ii) memorisation in parts at the granularity of individual tokens, but not necessarily in the order in which they appear in the string. Next, we investigate memorisation mechanics by checking to what extent different parts of a token’s prefix in the string are necessary and sufficient to recollect the token. We leverage our insights to explain the dynamics of memorising strings and we conclude by discussing the implications of our findings for quantifying memorisation.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9281