Training pansharpening networks at full resolution using degenerate invariance

YiChang Qu, Bing Li, Jie Huang, Feng Zhao

Abstract: Pan-sharpening is an important technique for remote sensing imaging systems to obtain high resolution multispectral images. Existing deep learning-based methods mostly rely on using pseudo-groundtruth multi-spectral images for supervised learning. The whole training process only remains at the scale of reduced resolution, which means that the impact of the degradation process is ignored and high-quality images cannot be guaranteed at full resolution. To address the challenge, we propose a new unsupervised framework that does not rely on pseudo-groundtruth but uses the invariance of the degradation process to build a consistent loss function on the original scale for network training. Specifically, first, we introduce the operator learning method to build an exact mapping function from multi-spectral to panchromatic images and decouple spectral features and texture features. Then, through joint training, operators and convolutional networks can learn the spatial degradation process and spectral degradation process at full resolution, respectively. By introducing them to build consistency constraints, we can train the pansharpening network at the original full resolution. Our approach could be applied to existing pansharpening methods, improving their usability on original data, which is matched to practical application requirements. The experimental results on different kinds of satellite datasets demonstrate that the new network outperforms state-of-the-art methods both visually and quantitatively.

Primary Subject Area: [Content] Multimodal Fusion

Secondary Subject Area: [Experience] Multimedia Applications

Relevance To Conference: The main content of our work is to fuse two different modes of remote sensing multimedia data, extract the effective information contained in them and display it in a more intuitive and natural way, which provides a powerful help for people's understanding and further processing of remote sensing images. At the same time, our proposed framework is an innovative method of image multi-modal fusion, which may bring inspiration to related fields.

Submission Number: 2153