The Dual Nature of Plasticity Loss in Deep Continual Learning: Dissection and Mitigation

Haoyu Wang, Wei P Dai, Jiawei Zhang, Jialun Ma, Mingyi Huang, Yuguo Yu

Keywords: Deep Continual Learning, Loss of Plasticity, Finite-time Lyapunov Exponent

Abstract:

Loss of plasticity (LoP) is the primary cause of cognitive decline in normal aging brains next to cell loss.
Recent works show that similar LoP also plagues neural networks during deep continual learning (DCL).
While it has been shown that random perturbations of learned weights can alleviate LoP, its underlying mechanisms remain insufficiently understood.
Here we offer a unique view of LoP and dissect its mechanisms through the lenses of an innovative framework combining the theory of neural collapse and finite-time Lyapunov exponents (FTLE) analysis.
We show that LoP actually consists of two contrasting types: (i) type-1 LoP is characterized by highly negative FTLEs, where the network is prevented from learning due to the collapse of representations; (ii) while type-2 LoP is characterized by excessively positive FTLEs, where the network can train well but the growingly chaotic behaviors reduce its test accuracy.
Based on these understandings, we introduce Generalized Mixup, designed to relax the representation space for prolonged DCL and demonstrate its superior efficacy vs. existing methods.

Primary Area: Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)

Submission Number: 5431