A Cross-Modal Feature Fusion Method to Diagnose Macular Fibrosis in Neovascular Age-Related Macular Degeneration

Xin Lei, Zhongping Chen, Hong Liu, Juan Chen, Huanhuan Tan, Weiwei Dai, Xiangdong Wang, Heping Xu

Abstract: This paper presents a novel Cross-modal Fusion Method (CrossFM) for automated classification of macular fibrosis in neovascular Age-related Macular Degeneration (nAMD). Firstly, a CoTransNet network is proposed to combine convolution and attention blocks for exploring the low-frequency global and high-frequency local information of lesion region for each modality. Then a dual-branch CoTransNet network is adopted to accept input data from randomly paired Optical Coherence Tomography (OCT) and Ultra-Wide-angle Fundus (UWF) images of the same eye in each single branch. Secondly, we propose a cross-modal attention strategy to integrate characteristics across various modalities to optimize lesion descriptive abilities with dynamically adaptive weights for each branch. Finally, image-level results are aggregated to obtain eye-level diagnosis results using high-probability reliability maps. Experimental results verify the effectiveness of the proposed method and even to the level of senior ophthalmologists. CrossFM also outperforms some state-of-the-art multimodal methods on both constructed and public datasets.