Efficient Variational Continual Learning with optimal parameter trajectory in parameter hyperspace
Abstract: Continual learning, a foundational challenge in machine learning, grapples with critical issues of efficient parameter storage and robust regularization, especially in Bayesian neural networks. Our research addresses these challenges with significant contributions. To address the storage complexity of fully connected layer parameters, we propose an efficient method that substantially reduces memory requirements. In convolutional neural networks, tailored parameter storage for Bayesian networks becomes essential, countering the parameter escalation caused by uncertainty inclusion. In variational continual learning, our work introduces an enhanced regularization term that preserves Kullback-Leibler divergence strengths while overcoming associated challenges. We also present an augmented Evidence Lower Bound term, crucial for capturing correlations between data and network parameters. This enables the storage of common and distinctive parameter hyperspace bases, vital in continual learning. Our approach strategically divides the parameter subspace into common and distinctive subspaces, with conditions for effective backward and forward knowledge transfer, elucidating the network-parameter dataset correspondence. In summary, our contributions advance efficient and effective continual learning, offering insights into parameter storage and regularization techniques for Bayesian neural networks.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Russell_Tsuchida1
Submission Number: 1991
Loading