Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning

Keywords: Sparse Memory Finetuning, Memory Layers, Parameter-Efficient Finetuning, Catastrophic Forgetting, Continual Learning, Plasticity-Stability Tradeoff

Abstract:

Adapting a pretrained language model to a new task often hurts the
general capabilities it already had, a problem known as catastrophic
forgetting. Sparse Memory Finetuning (SMF) tries to avoid this by
adding key-value memory layers to the model and, on each training
step, updating only the small set of memory rows that the current
batch reads most heavily. We re-implement SMF on
Qwen-2.5-0.5B-Instruct and compare it with LoRA and full
finetuning on MedMCQA, a 4-choice medical exam task, using WikiText
perplexity and TriviaQA accuracy as forgetting probes. SMF improves
MedMCQA by 2.5 percentage points while keeping both forgetting
probes within roughly 1 point of the base model, whereas LoRA and
full finetuning achieve larger gains but with clear drift on both. We
also compare two row-selection rules (KL-divergence and TF-IDF), which
balance the two forgetting metrics differently.

Submission Number: 23