Learning Cross-Spectral Prior for Image Super-Resolution

Chenxi Ma, Weimin Tan, Shili Zhou, Bo Yan

Abstract: With the rising interest in multi-camera cross-spectral systems, cross-spectral images have been widely used in computer vision and image processing. Therefore, an effective super-resolution (SR) method is significant in providing high-resolution (HR) cross-spectral images for different research and applications. However, existing SR methods rarely consider utilizing cross-spectral information to assist the SR of visible images and cannot handle the complex degradation (noise, high brightness, low light) and misalignment problem in low-resolution (LR) cross-spectral images. Here, we first explore the potential of using near-infrared (NIR) image guidance for better SR, based on the observation that NIR images can preserve valuable information for recovering adequate image details. To take full advantage of the cross-spectral prior, we propose a novel $\textbf{C}$ross-$\textbf{S}$pectral $\textbf{P}$rior guided image $\textbf{SR}$ approach ($\textbf{CSPSR}$). Concretely, we design a cross-view matching (CVM) module and a dynamic multi-modal fusion (DMF) module to enhance the spatial correlation between cross-spectral images and to bridge the multi-modal feature gap, respectively. The DMF module facilitates adaptive feature adaptation and effective information transmission through a dynamic convolution and a cross-spectral feature transfer (CSFT) unit. Extensive experiments demonstrate the effectiveness of our CSPSR, which can exploit the prominent cross-spectral information to produce state-of-the-art results.

Primary Subject Area: [Content] Media Interpretation

Secondary Subject Area: [Content] Media Interpretation

Relevance To Conference:

This work proposes a cross-spectral image super-resolution method by utilizing cross-spectral information to handle the complex degradation (noise, high brightness, low light) and misalignment problem in low-resolution (LR) cross-spectral images.
Based on the observation that NIR images can preserve valuable information for recovering adequate image details, our method can effectively exploit the near-infrared (NIR) image information to guide the SR. Extensive experiments demonstrate the state-of-the-art performance of our method.

Submission Number: 2847