A General Cooperative Optimization Driven High-Frequency Enhancement Framework for Multispectral Image Fusion

Chentong Huang, Junming Hou, Chenxu Wu, Xiaofeng Cong, Man Zhou, Junling Li, Danfeng Hong

Abstract: Pan-sharpening essentially to boost the spatial resolution of a multispectral (MS) image guided by its paired panchromatic (PAN) image. In other words, this process intricately integrates the high-frequency components extracted from texture-rich PAN images into the low-resolution (LR) MS images, resulting in texture-rich MS images. Though existing deep learning (DL)-based techniques have made impressive performance compared with traditional algorithms, they still face challenges in accurately restoring high-frequency details in MS images, thus limiting overall pan-sharpening performance. In addition, reference high-resolution (HR) MS images are often underutilized, typically serving only as training labels. In this work, we present a general high-frequency enhancement framework for pan-sharpening, which is implemented through a cooperative optimization strategy using mutual information (MI) maximization and contrastive learning. Specifically, our model comprises two fundamental modules: the high-frequency feature alignment (HFFA) module and the high-frequency detail calibration (HFDC) module. The first employs MI maximization to align the high-frequency semantic statistical distribution between PAN images and reference HRMS images. The latter is designed to calibrate the high-frequency components of MS modality under the guidance of the PAN counterparts through the contrastive learning constraint, thereby producing more accurate high-frequency information on MS modality. By integrating the calibrated high-frequency features of MS modality and those of PAN modality, we can obtain a more comprehensive and precise high-frequency feature representation of these two modalities, facilitating the reconstruction of LRMS images. Our model, incorporating the aforementioned key elements, significantly surpasses other state-of-the-art (SOTA) techniques across multiple satellite datasets in both quantitative and qualitative experiments. Moreover, the real-world full-resolution and cross-sensor assessments testify to its exceptional generalization capabilities. The code is available at https://github.com/Vcocoi/CONet.