Towards Continual Domain Adaptation of Vision-language Models

Yilei Jiang, Yiyuan Zhang, Kaixiong Gong, Jiaming Han, Wei-Hong Li, Xiangyu Yue

Keywords: continual learning, domain adaptation, vision-language models

Abstract: Large-scale vision-language models have achieved remarkable performance on various downstream tasks. {Nevertheless, how to efficiently adapt vision-language models to new data distributions without re-training, \ie, domain incremental learning (DIL) of vision-language models, is still under-explored. Existing DIL methods for single modality are either not applicable to multi-modal settings or need exemplar buffers to store previous samples to avoid catastrophic forgetting, which is not memory-efficient.} To address these limitations, we propose an exemplar-free paradigm to improve DIL of vision-language models based on prompt-tuning. We theoretically analyze and decompose the problem into two optimization objectives. Guided by the theoretical insights, we propose a novel framework named {M}ultimodal {C}ontinual {D}omain {A}daptation (MCDA), which incorporates two strategies: Multimodal Domain Alignment (MDA) and Maximum Softmax Gating (MSG). MDA enhances cross-domain performance by aligning visual and language representation spaces, while MSG improves the accuracy of domain identification by gating through Softmax probability. Extensivev experimental results demonstrate that our method outperforms current state-of-the-art approaches.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6831