Dynamic Knowledge Adapter with Probabilistic Calibration for Generalized Few-Shot Semantic Segmentation

Jintao Tong, Haichen Zhou, Yicong Liu, Yiman Hu, Yixiong Zou

Abstract: Generalized Few-shot Semantic Segmentation (GFSS) aims to use a few novel-class samples to enable the model trained on base classes to have the ability to segment for all classes (including base and novel classes). We analyze the three main reasons for the model’s limited performance on GFSS: the lack of adaptability to learn novel classes, the instability that causes the catastrophic forgetting of base classes, and the biased prediction of imbalanced classes. To handle these issues, we design an auxiliary network (Dynamic Knowledge Adapter, DKA) for the GFSS task. Firstly, DKA handles the adaptability problem by selecting only efficient parameters for finetuning. Secondly, DKA addresses the stability problem by relabelling part of the training samples for iterative training, which alleviates the conflict between base and novel classes. Thirdly, it involves a probabilistic calibration module to help the model rectify the prediction bias caused by imbalanced data. Experimental results show that these designs can help the model to take into account the segmentation performance of base classes, novel classes, and the background class, that is, to perform well in all-class segmentation.