Elastic Mean-Teacher Distillation Mitigates the Continual Learning Stability Gap

Albin Soutif, Joost van de weijer, Antonio Carta

Keywords: continual learning, continual evaluation

Abstract: Nowadays, neural networks are being used to solve a variety of tasks. They are very effective when trained on large datasets. However, in continual learning, they are trained on non-stationary stream of data, which often results in forgetting of the previous knowledge. In the literature, continual learning models are exposed to a sequence of tasks, and must learn each task one by one. They are then evaluated at the end of each learning session. This allows to measure the average accuracy over all tasks encountered so far. Recently De Lange et al. (2022) showed that continual learning methods suffer from the Stability Gap, encountered when evaluating the model continually. Even when the performance at the end of training is high, the worst-case performance is low, which could be a problem in applications where the learner needs to always perform greatly on all tasks while learning the new task. In this paper, we propose to apply a refined variant of knowledge distillation, adapted to the class-incremental learning setting, and used in combination with replay, to improve the stability of the continual learning algorithms. We also propose to use a distillation method derived from the Mean teacher distillation training paradigm introduced in semi-supervised learning. We demonstrate empirically that the use of this method enhances the stability in the more challenging setting of online continual learning.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning