A Novel Multi-scale Feature Fusion Based Network for Hyperspectral and Multispectral Image Fusion

Shuai Dong, Shaoguang Huang, Jinhan Zhang, Hongyan Zhang

Abstract: Fusion of low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (HR-MSI) has become an effective technique for HSI super-resolution. Deep leraning based fusion methods have achieved significant success in the fields. However, they often show limited ability to capture the complex spatial and spectral information of HSI, resulting in the loss of details. In this paper, we develope a novel multi-scale feature fusion based network (MSFNet) for HSI super-resolution, which consists of a multi-scale feature extraction block and a multi-scale feature fusion block. In the former block, we fully consider the spatial and spectral correlations and develop two modules, i.e., global-local attention and channel self-attention, to capture the complex structure of HSI at different scales. In the fusion stage, we adopt a U-Net like architecture to gradually fuse the extracted multi-scale features, resulting in restored HSIs at different scales. We also develop a new loss function to train the proposed neural network by minmizing the restoration errors at different scales both in the raw domain and the frequency domain, which facilitates to preserve the high-frequency details. Our experimental results demonstrate that the proposed model outperforms the state-of-the-art.