Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation

Jintao Tong, Ran Ma, Yixiong Zou, Guangyao Chen, Yuhua Li, Ruixuan Li

TL;DR: We find the adapter not only helps the fine-tuning of downstream tasks but also naturally acts as a domain decoupler. Based on this, we proposed a structure-based decoupling method to bridge domain gap, achieving superior performance in CD-FSS.

Abstract: Cross-domain few-shot segmentation (CD-FSS) is proposed to first pre-train the model on a source-domain dataset with sufficient samples, and then transfer the model to target-domain datasets where only a few training samples are available for efficient finetuning. There are majorly two challenges in this task: (1) the domain gap and (2) finetuning with scarce data. To solve these challenges, we revisit the adapter-based methods, and discover an intriguing insight not explored in previous works: the adapter not only helps the fine-tuning of downstream tasks but also naturally serves as a domain information decoupler. Then, we delve into this finding for an interpretation, and we find the model's inherent structure could lead to a natural decoupling of domain information. Building upon this insight, we propose the Domain Feature Navigator (DFN), which is a structure-based decoupler instead of loss-based ones like current works, to capture domain-specific information, thereby directing the model's attention towards domain-agnostic knowledge. Moreover, to prevent the potential excessive overfitting of DFN during the source-domain training, we further design the SAM-SVN method to constrain DFN from learning sample-specific knowledge. On target domains, we freeze the model and fine-tune the DFN to learn knowledge specific to target domains. Extensive experiments demonstrate that our method surpasses the state-of-the-art method in CD-FSS significantly by 2.69% and 4.68% average MIoU in 1-shot and 5-shot scenarios, respectively.

Lay Summary: Cross-domain few-shot segmentation (CD-FSS) aims to adapt models trained on one domain with ample data to new domains with only a few labeled examples. This task is challenged by domain shifts and the difficulty of fine-tuning with limited data. In this work, we revisit adapter-based methods and highlight a novel perspective: adapters not only facilitate efficient fine-tuning but also inherently decouple domain-specific information. Building on this insight, we propose the Domain Feature Navigator (DFN) to explicitly guide the model toward learning domain-agnostic representations. To prevent overfitting to source-domain biases during training—which could hinder adaptation—we further introduce SAM-SVN, a regularization technique that limits the DFN from capturing sample-specific noise. Together, DFN and SAM-SVN enable better generalization and more effective adaptation in CD-FSS settings. Experiments show that our method significantly outperforms prior approaches in both 1-shot and 5-shot segmentation scenarios.

Primary Area: General Machine Learning->Transfer, Multitask and Meta-learning

Keywords: Decouple, Cross-Domain, Few-Shot, Segmentation

Submission Number: 8891