Multi-modality medical image fusion by edge supervising and multi-scale attention features extraction

Wencheng Mei, Kangjian He, Dan Xu, Siqi Xie, Yiqiao Zhou

Abstract: Since the development of deep learning, multimodal medical image fusion (MMIF) has achieved both efficiency and real-time performance. However, most existing deep learning-based fusion methods primarily focus on the overall network architecture, often overlooking the intrinsic characteristics of the source images. From the perspectives of SPECT and PET imaging and the continuity of biological information, the edge regions of these functional images should receive greater attention during the fusion process. Furthermore, the pseudocolor should be separated before fusion and reintroduced afterward. Since functional images such as SPECT and PET typically suffer from low clarity, directly fusing them without proper processing may obscure texture details in the resulting image. To address these challenges, we propose an end-to-end encoder–decoder network for multimodal medical image fusion, termed EFCNet. The encoder comprises three main components: a smooth edge extraction module (SEEM), a multi-scale attention module (MSAM), and the E-Fusion module. The decoder reconstructs the fused image from these features. Specifically, SEEM extracts and smooths the edge information of functional source images (SPECT and PET), thereby mitigating the issues mentioned above. MSAM captures both local details and global contextual features while adaptively emphasizing more informative channels. E-Fusion then performs effective fusion of the extracted local and global features. Notably, our model is trained on a single dataset to obtain the pretrained weights, yet it achieves impressive results when tested on other datasets, demonstrating the strong generalization capability of the proposed method. The implementation of our proposed method is available on GitHub at https://github.com/VCMHE/EFCNet.

External IDs:dblp:journals/tjs/MeiHXXZ25