Forgetting of task-specific knowledge in model merging-based continual learning

Timm Hess, Gido M van de Ven, Tinne Tuytelaars

Keywords: continual learning, model merging, computer vision

TL;DR: Linearly merging models in continual learning preserves shared knowledge but degrades unshared specifics, creating a trade-off based on whether lost information was a useful skill or unintended bias.

Abstract: This paper investigates the linear merging of models in the context of continual learning (CL). Using controlled visual cues in computer vision experiments, we demonstrate that merging largely preserves or enhances shared knowledge, while unshared task-specific knowledge rapidly degrades. We further find that merging models from an incremental training process consistently outperforms merging models trained in parallel.

Submission Number: 7