Mining Shallow Layer Representations in Class-Incremental Learning

Bowen Zheng, Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan

Keywords: class-incremental learning

Abstract:

Class-Incremental Learning (CIL) aims to learn new knowledge without forgetting the old knowledge.
One of the popular approaches is to obtain transferable representations, which would be general for learning incremental tasks without expanding the representations.
Recently, many works focus on making the final representation more transferable across incremental tasks.
However, researchers rarely focus on shallow layer representations and utilize their properties to facilitate CIL, although they are shown to be more transferable than the final representation.
In this paper, we investigate the properties of the shallow layer representations and utilize them to improve the performance in class-incremental learning.
Specifically, we show that shallow layer representations forget less than deeper layers.
Furthermore, we find that shallow layer representations have more stable intra-class relations.
Such intra-class relations reflect the task-agnostic information that the deeper layer representations lack.
Therefore, we propose Intra-class Backward Distillation (IncBD) to make the deeper layers learn from the intra-class relations of the shallow layer's representations, making the final representation more stable in terms of the intra-class relations.
To compensate for the loss of class separability introduced by backward distillation, we also propose to train auxiliary classifiers for each layer's representation.
Extensive experiments are performed to show that the intra-class relations are important for the transferability of the final representation and performance improvement in class-incremental learning.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3108