DP-INNet: Dual-Path Implicit Neural Network for Spatial and Spectral Features Fusion in Pan-Sharpening

Jingjia Huang, Ge Meng, Yingying Wang, Yunlong Lin, Yue Huang, Xinghao Ding

Abstract: Pan-sharpening is a technique that fuses a high-resolution panchromatic (PAN) image with its corresponding low-resolution multispectral (MS) image to create a high-resolution multispectral image. Due to the powerful representation ability of Convolutional Neural Networks (CNNs), deep learning-based pan-sharpening methods have rapidly developed in recent years. However, existing methods often ignore the representation of multimodal information from the perspective of continuous physical signals, which inevitably leads to the loss of detailed information during the fusion process. Therefore, this paper proposes a novel pan-sharpening method that integrates spectral information with structural information in a continuous domain by using implicit neural representation (INR). Specifically, an implicit representation function is used to align the spatial information of multimodal images in the continuous domain, which preserves structural details. Additionally, a gated convolutional network is utilized to achieve interaction between different order spectral information in multispectral images. Finally, an MLP is used to fuse the spatial and spectral information in the continuous space to generate the expected high-resolution multispectral image. Extensive experiments on different datasets show that our method outperforms existing methods in terms of both quantitative and qualitative metrics.