Blessing Few-Shot Segmentation via Semi-Supervised Learning with Noisy Support Images

Runtong ZHANG, Hongyuan Zhu, Hanwang Zhang, Chen Gong, Joey Tianyi Zhou, Fanman Meng

Abstract: Mainstream few-shot segmentation methods meet performance bottleneck due to the data scarcity of novel classes with insufficient intra-class variations, which results in a biased model primarily favoring the base classes. Fortunately, owing to the evolution of the Internet, an extensive repository of unlabeled images has become accessible from diverse sources such as search engines and publicly available datasets. However, such unlabeled images are not a free lunch. There are noisy inter-class and intra-class samples causing severe feature bias and performance degradation. Therefore, we propose a semi-supervised few-shot segmentation framework named \textbf{F4S}, which incorporates a ranking algorithm designed to eliminate noisy samples and select superior pseudo-labeled images, thereby fostering the improvement of few-shot segmentation within a semi-supervised paradigm. The proposed F4S framework can not only enrich the intra-class variations of novel classes during the test phase, but also enhance meta-learning of the network during the training phase. Furthermore, it can be readily implemented with ease on any off-the-shelf few-shot segmentation methods. Additionally, based on a Structural Causal Model (SCM), we further theoretically explain why the proposed method can solve the noise problem: the severe noise effects are removed by cutting off the backdoor path between pseudo labels and noisy support images via causal intervention. On PASCAL-5$^{i}$ and COCO-20$^{i}$ datasets, we show that the proposed F4S can boost various popular few-shot segmentation methods to new state-of-the-art performances.