Learning to Generalize Heterogeneous Representation for Cross-Modality Image Synthesis via Multiple Domain Interventions

Yawen Huang, Huimin Huang, Hao Zheng, Yuexiang Li, Feng Zheng, Xiantong Zhen, Yefeng Zheng

Abstract: Magnetic resonance imaging with modality diversity substantially increases productivity in routine diagnosis and advanced research. However, high inter-equipment variability and expensive examination cost remain as key challenges in acquiring and utilizing multi-modal images. Missing modalities often can be synthesized from existing ones. While the rapid growth in image style transfer with deep models overwhelms the above endeavor, such image synthesis may not always be achievable and even impractical when applied to medical data. The proposed method addresses this issue by a convolutional sparse coding (CSC) adaptation network to handle the lacking of generalizing medical image representation learning. We reduce both inter-domain and intra-domain divergences by the domain-adaptation and domain-standardization modules, respectively. On the basis of CSC features, we penalize their subspace mismatching to reduce the generalization error. The overall framework is cast in a minimax setting, and the extensive experiments show that the proposed method yields state-of-the-art results on multiple datasets.