Cross-Modality Interaction Network for Pan-Sharpening

Yingying Wang, Xuanhua He, Yuhang Dong, Yunlong Lin, Yue Huang, Xinghao Ding

Abstract: Pan-sharpening seeks to generate a high-resolution multispectral (HRMS) image by merging the high-resolution panchromatic (PAN) image and its low-resolution multispectral (LRMS) counterpart. The main challenge lies in enhancing modality-aware features and efficiently integrating complementary information between PAN and MS pairs. To achieve desired fusion results, it is crucial to fully utilize both intramodality characteristics and intermodality relationships. Current research often overlooks the exploration of cross-modality relationships and neglects the enhancement of modality-aware features in pan-sharpening. In this work, we introduce an innovative pan-sharpening framework, named cross-modality interaction network (CMINet), which comprises three core designs: a modality-aware feature enhancement (MAFE) module to enhance the feature representation of both modalities, a cross-modality attention (CMA) module that effectively extracts the intramodality features and fully leverages the intermodality complementary information, and a modality alignment (MA) module to address modality-aware misalignment issue during fusion. Extensive experiments are conducted to verify the effectiveness of our proposed network and showcase its superior performance in comparison to other state-of-the-art approaches.