Continual Learning of Foundation Models with Limited Labelled Data
Keywords: Continual learning, few-shot learning, foundation model
Abstract: We explore a new paradigm of continual learning dubbed Few-Shot Class-Incremental Tuning (FSCIT), which facilitates continual tuning of vision foundation models to continuously learn new classes with few samples per class. Unlike traditional Few-Shot Class-Incremental Learning (FSCIL), FSCIT does not assume the availability of a large in-distribution base session to initially train the model in a fully supervised setting, prior to the few-shot class-incremental sessions. To this end, we propose Consistency-guided Asynchronous Contrastive Tuning (CoACT), a new approach to continually tune foundation models for new classes in few-shot settings. CoACT comprises three components: (i) asynchronous contrastive tuning, which learns new classes by including LoRA modules in the pre-trained encoder while enforcing consistency between two asynchronous encoders; (ii) controlled fine-tuning, which facilitates effective tuning of a subset of the foundation model; and (iii) consistency-guided incremental tuning, which enforces additional regularization during later sessions to reduce forgetting of the learned classes. We perform an extensive study on 16 diverse datasets where CoACT outperforms the best baseline method by 2.47% on average and with up to 12.52% on individual datasets. Additionally, CoACT shows reduced forgetting and robustness in low-shot experiments. As an added bonus, CoACT outperforms current SOTA on FSCIL.
Submission Number: 2
Loading