Memory-Based Contrastive Learning with Optimized Sampling for Incremental Few-Shot Semantic Segmentation

Yuxuan Zhang, Miaojing Shi, Taiyi Su, Hanli Wang

Abstract: Incremental few-shot semantic segmentation (IFSS) aims to incrementally expand a semantic segmentation model’s ability to identify new classes based on few samples. However, it grapples with the dual challenges of catastrophic forgetting (due to feature drift in old classes) and overfitting (triggered by inadequate samples in new classes). To address these issues, a novel approach is proposed to integrate pixel-wise and region-wise contrastive learning, complemented by an optimized example and anchor sampling strategy. The proposed method incorporates a region memory and pixel memory designed to explore the high-dimensional embedding space more effectively. The memory, retaining the feature embeddings of known classes, facilitates the calibration and alignment of seen class features during the learning process of new classes. To further mitigate overfitting, the proposed approach implements an optimized example and anchor sampling strategy. Extensive experiments show the competitive performance of the proposed method. The source code of this work can be found in https://mic.tongji.edu.cn.