Domain Prompt Matters a Lot in Multi-Source Few-Shot Domain Adaptation
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Large language-vision models, few-shot learning, domain adaptation, mixup
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: Large vision-language models have demonstrated strong performance in multi-source few-shot domain adaptation (MFDA). Current methods predominantly like CoOp rely on identifying a domain-agnostic prompt, leading to the overlooking of known difference information between domains. However, extracting the domain information requires the model to have good identification ability for domain information. Although training models with domain prompts allow them to capture the specific semantic nuances of a particular domain, using learnable prompts increases the risk of overfitting on training samples and reduces the effectiveness of domain prompts in capturing target domain features during transfer. To address this challenge, we propose "domain-aware mixup," a method that allows the model to become more sensitive to specific domain information when facing cross-domain mixed feature information. Specifically, we design the prompt structure composed of domain prompt and context prompt to narrow the gap between the specific domain feature and the specific image feature extracted from the cross-domain mix feature. This approach enables us to efficiently train domain prompt terms, enhancing the model's ability to distinguish semantic distinctions between different domains. We empirically validate our method on the DomainNet and OfficeHome datasets, observing a performance boost of 5.3%-5.8% over the CLIP model and a 1.1%-1.5% advantage over the domain-agnostic tuning method.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 4781
Loading