Rethinking the Role of Verbatim Memorization in LLM Privacy

Tom Sander, Bargav Jayaraman, Mark Ibrahim, Kamalika Chaudhuri, Chuan Guo

Keywords: LLM;memorization;privacy

TL;DR: Increased verbatim memorization doesn't necessarily lead to greater chat extractability, and model quality is a greater privacy threat than memorization

Abstract:

Conventional wisdom in machine learning privacy research states that memorization
directly implies a loss of privacy. In contrast, a well-generalized model only remembers distributional patterns and preserves privacy of its training data. In this work, we show that this relationship is much more complex for LLMs trained for chat,
and depends heavily on how knowledge is encoded and manipulated. To this end, we fine-tune language models on synthetically generated biographical information including PIIs, and try to extract them in different ways after instruction fine-tuning. We find counter to conventional wisdom that better verbatim memorization does not necessarily increase data leakage via chat. We also find that it is easier to extract information via chat from an LLM that is better able to manipulate and process knowledge even if it is smaller, and that not all attributes are equally extractable. This suggests that the relationship between privacy, memorization and language understanding of LLMs is very intricate, and that examining memorization in isolation can lead to misleading conclusions.

Primary Area: Social and economic aspects of machine learning (e.g., fairness, interpretability, human-AI interaction, privacy, safety, strategic behavior)

Submission Number: 20145