Continual Memorization of Factoids in Language Models

TMLR Paper5123 Authors

Abstract: As new knowledge rapidly accumulates, language models (LMs) with pretrained knowledge quickly become obsolete. A common approach to updating LMs is fine-tuning them directly on new knowledge. However, recent studies have shown that fine-tuning for memorization may be ineffective in storing knowledge or may even exacerbate hallucination—raising doubts about its reliability when applied repeatedly. To study this, we formalize the problem of continual memorization, where a model must memorize and retain a set of factoids through multiple stages of fine-tuning on subsequent datasets. We first characterize the forgetting patterns through extensive experiments and show that LMs widely suffer from forgetting, especially when needing to memorize factoids in the second stage. We posit that forgetting stems from suboptimal training dynamics which fails to: (1) protect the memorization process when learning factoids or (2) reduce interference from subsequent training stages. To test this hypothesis, we explore various data mixing strategies to alter the fine-tuning dynamics. Intriguingly, we find that mixing randomly generated word sequences or generic data sampled from pretraining corpora at different training stages effectively mitigates forgetting (REMIX: Random and Generic Data Mixing). REMIX can recover performance from severe forgetting, outperforming replay methods and other continual learning baselines. We analyze how data mixing can influence the learning process and find that robust memorization follows a distinct pattern—the model stores factoids in earlier layers than usual and diversifies the layers that retain them, which results in easier recall and manipulation of the learned factoids.

Submission Length: Regular submission (no more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=pMGj34nDi0&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DTMLR%2FAuthors%23your-submissions)

Changes Since Last Submission: Fixed the formatting issue.

Assigned Action Editor: ~Adina_Williams1

Submission Number: 5123