Learning Diffusion High-Quality Priors for Pan-Sharpening: A Two-Stage Approach With Time-Aware Adapter Fine-Tuning

Yingying Wang, Yunlong Lin, Xuanhua He, Hui Zheng, Keyu Yan, Linyu Fan, Yue Huang, Xinghao Ding

Abstract: Pan-sharpening aims to enhance the spatial resolution of the low-resolution multispectral (LRMS) image by incorporating high-frequency details from the panchromatic (PAN) image, while maintaining the spectral qualities of the LRMS image. Recent advancements in diffusion models have shown remarkable capabilities in image restoration and generation. However, simply applying diffusion models in pan-sharpening yields suboptimal outcomes in terms of fine-grained details and spectral fidelity. To this end, we introduce TA-DiffHQP, a two-stage approach that integrates the diffusion high-quality priors model (DiffHQP) and the time-aware adapter (TA-Adapter). Initially, we perform self-reconstruction pretraining DiffHQP with a fixed sampling strategy on approximately 24K high-resolution remote sensing datasets to explicitly model the high-quality texture details and spectral fidelity, after which we freeze most of DiffHQP’s parameters. In stage two, we integrate time-aware fusion adapters with the DiffHQP, enabling rapid adaptation to the pan-sharpening task. The TA-Adapters prioritize low-frequency main scenes during the early phases of the denoising process and refine high-frequency details in the later phases, achieving cross-modal information fusion from coarse to fine. Extensive experiments conducted on three satellite datasets demonstrate that our approach attains state-of-the-art (SOTA) performance over existing methods, revealing superior fusion outcomes in pan-sharpening.