Dual Spatial-Spectral Pyramid Network With Transformer for Hyperspectral Image Fusion

Yucheng Sun, Han Xu, Yong Ma, Minghui Wu, Xiaoguang Mei, Jun Huang, Jiayi Ma

Abstract: Multispectral image (MSI) and hyperspectral image (HSI) fusion can combine the best of both worlds to produce images with both high spatial and spectral resolutions. In this article, we have designed a network for fusing MSIs and HSIs, called DSPNet. On the one hand, in order to ensure the accuracy of the spectral dimension, i.e., spectral fidelity, we designed the spectral pyramid (SpePy) module and the multiscale local spectral information fusion (MLSIF) module. The former extracts the multiscale local spectral information that captures the subtle spectral details and variations between different spectra. The latter establishes long-range dependency in the spectral dimension through the spectralwise multihead hybrid-attention (S-MHA) mechanism, thus enabling the network to focus on the local spectral information needed to recover the spectral details. On the other hand, to address the spatial information of MSIs, we designed the spatial pyramid (SpaPy) module. The SpaPy module can extract the nonlocal spatial information of MSIs at different scales, which enables the network to adapt to different remote sensing scenes. Experiments performed on simulated and real data demonstrate the superiority of our method over the state-of-the-art methods both qualitatively and quantitatively. Our code is publicly available at https://github.com/syc11-25/DSPNet .