Preserving Representation In Continual Learning via Feature-Preserving Fine-Tuning

ICLR 2026 Conference Submission18690 Authors

Keywords: continual learning, representation learning, fine-tuning, foundation models

Abstract:

In real-world applications, deep learning models must continually adapt to sequentially arriving tasks without access to previous data. Although pre-trained foundation models show generalisation and zero-shot abilities, fine-tuning them in a continual learning setting often leads to representation degradation. In this study, we firstly systematically evaluate several recent feature-preserving fine-tuning methods (L2-SP, FTP, WiseFT and ImpReg) in continual learning scenario using a large scale pre-trained foundation model. We further explore the
effectiveness of full fine-tuning (FullFT) versus parameter-efficient fine-tuning (PEFT) and propose a novel two-stage fine-tuning strategy,
PEFT+Cons, designed to balance stability and plasticity by combining PEFT with task-specific knowledge consolidation. Extensive experiments on the CIFAR-100 and ImageNet-R benchmark datasets demonstrate that our proposed PEFT+Cons approach effectively prevents representation forgetting while enhancing task-specific knowledge retention.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 18690