POC: Preventing the Over-Collapse of Classes for Class-Incremental Learning

Shengchao Zhou, Bolin Ni, Youzhi Zhang, Zhaoxiang Zhang, XIAOJUAN QI, Gaofeng Meng

Keywords: Class-Incremental Learning, Over-Collapse, Catastrophic Forgetting

Abstract: Deep neural network-based classification models often suffer from catastrophic forgetting during class-incremental learning (CIL). Previous studies reveal that it results from the overlap between seen and future classes after being mapped by model to its feature space through extracting the features. In this paper, we analyze that this overlap mainly results from the $\textit{over-collapse}$ of seen classes, where the model tends to map originally separated one seen class and its adjacent regions in input space to be mixed in the feature space, making them indistinguishable. To this end, we propose a two-step framework to $\textbf{P}$revent the $\textbf{O}$ver-$\textbf{C}$ollapse (POC). During training, POC first learns and applies a set of transformations to the training samples of seen classes. Based on our theoretical analysis, the transformation results will locate in the adjacent regions of the seen classes in the input space so that we can let them represent the adjacent regions. Then, the model's optimization objective is modified to additionally classify between the seen classes and the adjacent regions, separating them in model's feature space so that preventing the over-collapse. To retain the model's generalization on the seen classes, a deterministic contrastive loss that makes the separate features of seen classes and adjacent regions close is further introduced. Since POC uses the adjacent regions exclusively for classification, it can be easily adopted by existing CIL methods. Experiments on CIFAR-100 and ImageNet demonstrate that POC effectively increases the last/average incremental accuracy of six SOTA CIL methods by 3.5\%/3.0\% on average respectively.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5728