Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation

Ido Nitzan Hidekel, Dan Raviv

Keywords: Catastrophic forgetting, Continual learning, Neural tangent kernel (NTK), Function-space analysis, Low-rank structure, Theoretical analysis

TL;DR: We derive a closed-form predictor for the catastrophic forgetting vector from NTK geometry, exact on frozen transformers. Forgetting concentrates in 1–6 eigenmodes; a Kronecker rule fixes the vulnerable rank as output-dimension times feature rank.

Abstract: Catastrophic forgetting is the defining obstacle to continual adaptation: it disrupts instruction-tuning and alignment maintenance in large language models, destabilizes domain-incremental vision systems under distribution shift, and limits the reuse of pretrained backbones across downstream tasks. Despite a decade of mitigation strategies such as parameter regularization, replay, gradient projection or functional distillation, none explains \emph{why} forgetting occurs or identifies the mechanism that produces it. We propose a function-space account in the Neural Tangent Kernel regime that derives the mechanism of forgetting mathematically: new-task training induces interference along a structured, low-rank subspace of output space defined by the cross-task NTK. A closed-form predictor identifies the forgetting vector - direction and magnitude jointly, with cosine similarity indistinguishable from $1$ on transformer backbones in the PEFT-CL regime. The theory further shows that forgetting concentrates in only 1-6 NTK eigenmodes and yields a Kronecker scaling rule for the vulnerable rank under frozen heads. These structural results explain why parameter-space methods fail on shared-head benchmarks and motivate a targeted spectral regularizer.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 10