Learning Continually at Peak Performance with Continuous Continual Backpropagation

Luc McCutcheon, Evangelos Chatzaroulas, Saber Fallah

Keywords: Continual Learning, Reinforcement Learning, Plasticity, Dormant Neurons, Optimizer, Continual Reinforcement Learning

TL;DR: We introduce Continuous Continual Backpropagation (CCBP), which introduces utility-scaled partial resets inplace of full neurons to maintain peak performance throughout Continual Reinforcement Learning

Abstract: Training neural networks under non-stationary data distributions, as in continual supervised and reinforcement learning, is hindered by loss of plasticity and representation collapse. While recent approaches employ periodic, full neuron reinitialization to sustain gradient flow and restore plasticity, they often sacrifice performance and still suffer frequent collapse. To address these limitations, we propose Continuous Continual Backpropagation (CCBP), which instead continuously, partially resets units. Empirically, CCBP outperforms both decay-based and reset-based methods after long sequences of distribution shifts, and uniquely prevents policy collapse in challenging continual reinforcement learning environments. Ablations further show how CCBP can be tuned to smoothly trade off plasticity and performance, highlighting gradual reinitialization as a promising direction for continual deep learning.

Primary Area: reinforcement learning

Submission Number: 24844