Fusformer: A Transformer-Based Fusion Network for Hyperspectral Image Super-Resolution

Jin-Fan Hu, Ting-Zhu Huang, Liang-Jian Deng, Hong-Xia Dou, Danfeng Hong, Gemine Vivone

Abstract: Hyperspectral image super-resolution (HISR) is to fuse a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI), aiming to obtain a high-resolution hyperspectral image (HR-HSI). Recently, various convolution neural network (CNN)-based techniques have been successfully applied to address the HISR problem. However, these methods generally only consider the relation of a local neighborhood by convolution kernels with a limited receptive field, thus ignoring the global relationship in a feature map. In this letter, we design a Transformer-based architecture (called Fusformer) for the HISR problem, which is the first attempt to apply the Transformer architecture to this task to the best of our knowledge. Because of the excellent ability of feature representations, especially by the self-attention (SA) in the Transformer, our approach can globally explore the intrinsic relationship within features. Considering the specific HISR problem, since the LR-HSI holds the primary spectral information, our method estimates the spatial residual between the upsampled low-resolution multispectral image (LR-MSI) and the desired HR-HSI, reducing the burden of training the whole data in a smaller mapping space. Various experiments show that our approach outperforms current state-of-the-art (SOTA) HISR methods. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/J-FHu/Fusformer</uri> .