Memory Adapters Enable Fast, Flexible Knowledge Unlearning in LLMs

Keltin Grimes, Kevin Kuo, Steven Wu, Virginia Smith, Marissa Catherine Connor

Keywords: Unlearning, LLM, memory layer, memorization

TL;DR: Memory adapters are fine-tunable adapters for LLMs that allow isolating sequence-level gradient updates to small modular sets of parameters, enabling efficient unlearning.

Abstract: We introduce memory adapters, a fine-tunable adapter for LLMs that allows isolating sequence-level gradient updates to small modular sets of parameters. The modularity of memory adapters enables instantaneous unlearning of any combination of documents, with empirically strong unlearning performance on the TOFU benchmark. Iterative unlearning is costless, and documents can even be flexibly included or excluded for individual queries in a batch. These unique properties are especially valuable for domains where data-removal requests may be unpredictable, granular, or user- or input-dependent.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 43