Multi-Source Domain Adaptation by Causal-Guided Adaptive Multimodal Diffusion Networks

Ziyun Cai, Yawen Huang, Tengfei Zhang, Yefeng Zheng, Dong Yue

Abstract: Multi-source domain adaptation (MSDA) strives to adapt the models trained on multimodal labelled source domains to an unlabelled target domain. Recent GANs based MSDA methods implicitly characterize the image distribution, which may result in limited sample fidelity, causing misalignment of pixel-level information among sources and the target. Furthermore, when samples from different sources interfere during the learning process, significant misalignment across different source domains may arise. In this paper, we propose a novel MSDA framework, called Causal-guided Adaptive Multimodal Diffusion Networks (C-AMDN), to tackle these challenges. C-AMDN incorporates a diffusive adversarial generation model for high-fidelity, efficient adaptation among source and target domains, along with deep causal inference re-weighting mechanism for the decision-making process that the conditional distributions of outcomes remain consistent across different domains, even as the input distributions change. In addition, we propose an efficient way to further adapt the input image to another domain: we preserve important semantic information by a density constraint regularization in the generation model. Experimental results demonstrate that C-AMDN significantly outperforms existing methods across several real-world domain adaptation benchmarks.