Predicting Task Forgetting in Large Language Models

Anat Kleiman, Jonathan Frankle, Sham M. Kakade, Mansheej Paul

Keywords: Large Language Models, Catastrophic Forgetting, Empirical, Finetuning, Deep Learning, Language Models, Forgetting

Abstract: In this paper, we offer a comprehensive evaluation of forgetting in large language models (LLMs) during sequential learning of finetuning tasks in a pretrained model. We empirically track the degradation of performance across diverse tasks and find that the validation perplexity can be predicted using a linear function, regardless of the specific task, model architecture, or task order. This knowledge sheds light on the dynamics of knowledge acquisition and retention, offering practical implications for managing and mitigating task forgetting in LLM-based systems.

Submission Number: 34