Is Class Incremental Learning Truly Learning Representations Continually?

Sungmin Cha, Dongsub Shim, Hyunwoo Kim, Moontae Lee, Honglak Lee, Taesup Moon

Keywords: continual learning, class-incremental learning, representation learning

Abstract:

Class incremental learning (CIL) aims to continually learn a classifier for new object classes from incrementally arriving data while not forgetting the past learned classes. The average test accuracy across all classes learned so far has been a widely used metric to evaluate the CIL algorithms, but we argue that a simple horse race toward maximizing the accuracy may not necessarily lead to developing effective CIL algorithms. Namely, since a classification model is often used as a backbone model that transfers the learned representations to other downstream tasks, we believe it is also important to ask whether the CIL algorithms are indeed learning representations continually. To that end, we borrow several typical evaluation protocols of representation learning to solely evaluate the quality of encoders learned by the CIL algorithms: 1) fix the encoder and re-train the final linear layer or run the k-nearest neighbor (NN) classifier using the entire training set obtained for all classes so far and check the test accuracy, and 2) perform transfer learning with the incrementally learned encoder to several downstream tasks and report the test accuracy on those tasks. Our comprehensive experimental results disclose the limitation of conventional accuracy-based CIL evaluation protocol as follows. First, the state-of-the-art CIL algorithms with high test accuracy do not necessarily perform equally well with respect to our representation-level evaluation, in fact, sometimes may perform even worse than naive baselines. Second, it turns out the high test accuracy of the state-of-the-art CIL algorithms may be largely due to the good quality of the representations learned from the first task, which means those algorithms mainly focus on stability (not forgetting the first task model's capability), but not really on continually learning new tasks, i.e., plasticity, to attain high overall average accuracy.
Based on these results, we claim that our representation-level evaluation should be an essential recipe for more objectively evaluating and effectively developing the CIL algorithms.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning