Wording Image for Domain-Invariant Representation in Domain Generalization
Keywords: Vision-Language Alignment, Domain Generalization, Domain-Invariant Representation Disentanglement, Long-Tail Learning
TL;DR: Learning domain-invariant embedding on joint vision-language embedding space for domain generalization without prior.
Abstract: The out-of-domain (OOD) generalization ability is essential for AI systems in real-world applications. Recent works proposed to utilize prior knowledge like unlabeled data or descriptions to improve the OOD performance. However, the assumption that the domain of each test data is known in advance is unrealistic in practice, which limits the generalization of AI system to various domains and prevents the wide deployment. In this paper, we introduce WIDIn, which Words the Images to learn Domain-Invariant features being robust to complex domain biases. Different from visual embeddings, the language embeddings of class descriptions are domain-invariant, and they can be connected via vision-language alignment. Thus, we propose to project images into language space by representing each image as a word token, which is attached with hand-crafted prompt and fed into language encoder. Then, the difference between the extracted embedding and the language embedding of its class description is used to estimate the domain-specific counterpart, which facilitates the domain-invariant representation learning. Notably, our WIDIn can be applied to both pretrained vision-language models like CLIP, and separately trained uni-modal models like MoCo and BERT. Experimental studies on two domain generalization benchmark datasets and two long-tail benchmark datasets demonstrate the effectiveness of our approach.
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6269
Loading