Multi-Modal Medical Image Fusion via 3D Manifold Fitting and Dual-Domain Cross-Attention

Zeyu Wang, Jiayu Wang, Haiyu Song, Jiawei Feng, Haoran Duan

Abstract: Medical image fusion (MIF) aims to extract complementary features from multi-modal source images and fuse them into a single image to assist in clinical diagnostics. Despite its importance, MIF faces two primary challenges: the lack of tailored paradigms for CMSF extraction and insufficient dual exploration of multi-modality and multi-frequency domains. To address these challenges, we propose a novel MIF model in this study. From the perspective of image manifolds, we reformulate CMSF extraction as a 3D manifold fitting problem and introduce a paradigm that uses mathematical fitting methods to generate CMSF. This approach achieves accurate feature extraction without the need for carefully designed loss functions as constraints, significantly reducing the number of parameters. Additionally, we introduce Cross-Modality Co-Frequency (CM-CoF) and Cross-Frequency Co-Modality (CF-CoM) attention modules, which explore implicit relationships between modalities and frequency domains. Experimental results demonstrate that the proposed model outperforms many state-of-the-art MIF algorithms.