Multimodal Parameter-Efficient Few-Shot Class Incremental Learning

Marco D'Alessandro, Alberto Alonso, Enrique Calabrés, Mikel Galar

Abstract:

Few-Shot Class Incremental Learning (FSCIL) is a challenging continual learning task, where limited training examples are available during several learning sessions. To
succeed in this task, it is necessary to avoid over-fitting
new classes caused by biased distributions in the few-shot
training sets. The general approach to address this issue
involves enhancing the representational capability of a predefined backbone architecture by adding special modules for
backward compatibility with older classes. However, this
approach has not yet solved the dilemma of ensuring high
classification accuracy over time while reducing the gap between the performance obtained on larger training sets and
the smaller ones. In this work, we propose an alternative
approach called Continual Parameter-Efficient CLIP (CPECLIP) to reduce the loss of information between different
learning sessions. Instead of adapting additional modules
to address information loss, we leverage the vast knowledge
acquired by CLIP in large-scale pre-training and its effectiveness in generalizing to new concepts. Our approach is
multimodal and parameter-efficient, relying on learnable
prompts for both the language and vision encoders to enable transfer learning across sessions. We also introduce
prompt regularization to improve performance and prevent
forgetting. Our experimental results demonstrate that CPECLIP significantly improves FSCIL performance compared
to state-of-the-art proposals while also drastically reducing
the number of learnable parameters and training costs.