Demystifying Language Model Forgetting with Low-Rank Example Associations
Keywords: Predicting Forgetting; Large Language Models
TL;DR: We analyze and predict associations between learned tasks and forgotten examples in LLMs
Abstract: Large Language models (LLMs) suffer from forgetting of upstream data when fine-tuned. Despite efforts on mitigating forgetting, few have investigated whether, and how forgotten upstream examples are associated with newly learned tasks. Insights on such associations enable efficient and targeted mitigation of forgetting. In this paper, we empirically analyze forgetting that occurs in $N$ upstream examples (of language modeling or instruction-tuning) after fine-tuning LLMs on one of $M$ new tasks, and visualize their associations with a $M\times N$ matrix. We empirically show that the degree of forgetting can often be approximated by simple multiplicative effects of the upstream examples and newly learned tasks. We also reveal more complicated patterns where specific subsets of examples are forgotten. Following our analysis, we predict forgetting that happens on upstream examples when learning a new task with matrix completion over the empirical associations, outperforming prior approaches that rely on trainable LMs. Replaying predicted examples can statistically significantly improve over random examples for alleviating forgetting.
Submission Number: 17
Loading