Rememory-Based SimSiam for Unsupervised Continual LearningDownload PDF

22 Sept 2022 (modified: 13 Feb 2023)ICLR 2023 Conference Withdrawn SubmissionReaders: Everyone
Keywords: Continual learning, unsupervised representation learning, contrastive learning, rememory process
TL;DR: We propose a novel rememory-based SimSiam method for unsupervised continual learning.
Abstract: Unsupervised continual learning (UCL) has started to draw attention from the continual learning community, motivated by the practical need of representation learning with unlabeled data on sequential tasks. However, most of recent UCL methods focus on mitigating the catastrophic forgetting problem with a replay buffer to store previous data (i.e., rehearsal-based strategy), which needs much extra storage and thus limits their practical applications. To overcome this drawback, based on contrastive learning via SimSiam, we propose a novel rememory-based SimSiam (RM-SimSiam) method to reduce the dependency on replay buffer under the UCL setting. The core idea of our RM-SimSiam is to store and remember the old knowledge with a data-free historical module instead of replay buffer. Specifically, this historical module is designed to store the historical average model of all previous models (i.e., the memory process) and then transfer the knowledge of the historical average model to the new model (i.e., the rememory process). To further improve the rememory ability of our RM-SimSiam, we devise an enhanced SimSiam-based contrastive loss by aligning the representations outputted by the historical and new models. Extensive experiments on three benchmarks demonstrate the effectiveness of our RM-SimSiam under the UCL setting.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics
Submission Guidelines: Yes
Please Choose The Closest Area That Your Submission Falls Into: Unsupervised and Self-supervised learning
7 Replies

Loading