YoooP: You Only Optimize One Prototype per Class for Non-Exemplar Incremental Learning

Jiangtao Kong, Zhenyu Zong, Tianyi Zhou, Huajie Shao

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: continual learning, non-exemplar incremental learning, prototype optimization

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Incremental learning (IL) usually addresses catastrophic forgetting of old tasks when learning new tasks by replaying old tasks' data stored in a memory, which can be limited by its size and the risk of privacy leakage. Recent non-exemplar IL methods only store class centroids as prototypes and perturb them with Gaussian noise to create synthetic data for replay. However, the class prototypes learned in different tasks might be close to each other, leading to the intersection of their synthetic data and forgetting. Moreover, the Gaussian perturbation does not preserve the real data distribution and thus can be detrimental. In this paper, we propose YoooP, a novel exemplar-free IL approach that can greatly outperform previous methods by only storing and replaying one prototype per class even without synthetic data replay. Instead of storing class centroids, YoooP optimizes each class prototype by (1) moving it to the high-density region within every class using an attentional mean-shift algorithm; and (2) minimizing its similarity to other classes' samples and meanwhile maximizing its similarity to samples from its class, resulting in compact classes distant from each other in the representation space. Moreover, we extend YoooP to YoooP+ by synthesizing replay data preserving the angular distribution between each class prototype and the class's real data in history, which cannot be obtained by Gaussian perturbation. YoooP+ effectively stabilizes and further improves YoooP without storing any real data. Extensive experiments demonstrate the superiority of YoooP/YoooP+ over non-exemplar baselines in terms of accuracy and anti-forgetting.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4662