CRNet: A Fast Continual Learning Framework With Random Theory

Depeng Li, Zhigang Zeng

Abstract: Artificial neural networks are prone to suffer from catastrophic forgetting. Networks trained on something new tend to rapidly forget what was learned previously, a common phenomenon within connectionist models. In this work, we propose an effective and efficient continual learning framework using random theory, together with Bayes’ rule, to equip a single model with the ability to learn streaming data. The core idea of our framework is to preserve the performance of old tasks by guiding output weights to stay in a region of low error while encountering new tasks. In contrast to the existing continual learning approaches, our main contributions concern (1) closed-formed solutions with detailed theoretical analysis; (2) training continual learners by one-pass observation of samples; (3) remarkable advantages in terms of easy implementation, efficient parameters, fast convergence, and strong task-order robustness. Comprehensive experiments under popular image classification benchmarks, FashionMNIST, CIFAR-100, and ImageNet, demonstrate that our methods predominately outperform the extensive state-of-the-art methods on training speed while maintaining superior accuracy and the number of parameters, in the class incremental learning scenario. Code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/toil2sweet/CRNet</uri> .