Choosing Wisely and Learning Deeply: Selective Cross-Modality Distillation via CLIP for Domain Generalization

Jixuan Leng, Yijiang Li, Haohan Wang

Abstract:

Domain Generalization (DG), a crucial research area, seeks to train models across multiple domains and test them on unseen ones. In this paper, we introduce a novel approach, namely, Selective Cross-Modality Distillation for Domain Generalization (SCMD). SCMD leverages the capabilities of large vision-language models, specifically CLIP, to train a more efficient model, ensuring it acquires robust generalization capabilities across unseen domains.
Our primary contribution is a unique selection framework strategically designed to identify hard-to-learn samples for distillation.
In parallel, we introduce a novel cross-modality module that seamlessly combines the projected features of the student model with the text embeddings from CLIP, ensuring the alignment of similarity distributions.
We assess SCMD's performance on various benchmarks, where it empowers a ResNet50 to deliver state-of-the-art performance, surpassing existing domain generalization methods. Furthermore, we provide a theoretical analysis of our selection strategy, offering deeper insight into its effectiveness and potential in the field of DG.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission: We have prepared the camera-ready version of the paper and would like to express our sincere gratitude to all the reviewers and action editors for their valuable feedback.

Code: https://github.com/SeanLeng1/SCMD

Assigned Action Editor: ~Hanwang_Zhang3

Submission Number: 1958