An Incremental Learning Approach for Sustainable Regional Isolation and Integration

Ziyi Wu, Hongge Yao, Jiaojiao Ma

Keywords: Continual Learning, Incremental Learning, Catastrophic Forgetting, Memory Replay, Regional Isolation, Regional Integration, Alleviate Recency Bias

TL;DR: Sustainable regional isolation and integration contribute to incremental learning while alleviating "recency bias".

Abstract: Humans are capable of acquiring new knowledge on a constant basis, while integrating and optimizing old knowledge without forgetting them. This is mainly attributed to the human brain’s ability of partitioned learning and memory replay. In this paper, we simulate this ability and propose an incremental learning network of Sustainable Regional Isolation and Integration (SRII). SRII consists of two phases, regional isolation and regional integration, which are iterated to achieve continuous incremental class learning. Regional isolation isolates new learning processes to avoid interfering with existing knowledge, while regional integration introduces knowledge distillation and margin loss regularization term, knowledge distillation to transfer replay knowledge for alleviating catastrophic forgetting, margin loss regularization term to clarify the boundaries of new and old knowledge for alleviating recency bias. Experimental results on the CIFAR100 and miniImageNet datasets demonstrate that SRII outperforms the state-of-the-arts to avoid catastrophic forgetting. In all 5-stage and 10-stage incremental settings, SRII outperforms the baseline and achieves at least $5.27\%+$ average accuracy improvement. Our source code is available at https://github.com/Wuziyi123/SRII.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning