Advancing Incremental Few-Shot Semantic Segmentation via Semantic-Guided Relation Alignment and Adaptation

Yuan Zhou, Xin Chen, Yanrong Guo, Jun Yu, Richang Hong, Qi Tian

Abstract: Incremental few-shot semantic segmentation aims to extend a semantic segmentation model to novel classes according to only a few labeled data, while preserving its segmentation capability on learned base classes. However, semantic aliasing between base and novel classes severely limits the quality of segmentation results. To alleviate this issue, we propose a semantic-guided relation alignment and adaptation method. Specifically, we first conduct semantic relation alignment in the base step, so as to align base class representations to their semantic information. Thus, base class embeddings are constrained to have relatively low semantic correlations to classes that are different from them. Afterwards, based on semantically aligned base classes, we further conduct semantic-guided adaptation during incremental learning, which aims to ensure affinities between visual and semantic embeddings of encountered novel classes, thereby making feature representations be consistent with their semantic information. In this way, the semantic-aliasing issue can be suppressed. We evaluate our model on PASCAL VOC 2012 and COCO datasets. The experimental results demonstrate the effectiveness of the proposed method.