Query Adaptive Transformer and Multiprototype Rectification for Few-Shot Remote Sensing Image Segmentation

Tianyi Gao, Zhi Gao, Hong Ji, Wei Ao, Weiwei Song

Abstract: Deep learning has emerged as a powerful tool for semantic segmentation tasks. However, in some data-deficient and resource-limited scenarios, networks are constrained to learn novel concepts. To tackle this problem, few-shot segmentation (FSS) has been proposed by the machine learning community, aiming at segmenting novel objects by leveraging a handful of annotated samples. However, most advanced FSS algorithms suffer from severe performance degradation when directly applied to remote sensing image (RSI) domains. Challenges arise due to the unique characteristics of RSIs. To address the large intraclass variation brought by various imaging conditions and large-scale variations, a query adaptive transformer (QAT) is proposed. QAT incorporates query priors into the feature extraction process, adapting RSI features to novel objects from query RSIs. It is noteworthy that query priors are extracted via a query prior generation (QPG) module devoid of any query label information. To alleviate interference brought by complex object distribution, a multiprototype strategy is adopted instead of representing objects with a single prototype ambiguously. Moreover, we rectify the prototypes using a prior injection module (PIM), thereby fully leveraging the advantages offered by query priors. The superiority of our method is validated through comprehensive experiments on the public iSAID- $5^{i}$ dataset and comparisons with state-of-the-art methods. Finally, we propose a novel cross-domain setting to investigate the potential and generalizability of few-shot RSI segmentation for several Earth observation applications.