Unsupervised Multi-Sensor Spectral Image Fusion via Frequency-Spatial Reciprocal-View Learning

Yuanchao Su, JinXu, Lianru Gao, Mengying jiang

Keywords: Unsupervised Learning, Multimodal Learning, Multi-View learning, Unsupervised Multi-Sensor Data Fusion, Multispectral and Hyperspectral Image Fusion

Abstract: The unsupervised fusion of multi-sensor spectral images is often limited by non-absolute registration. This misalignment leads to significant differences between the spectral shape of the fused image and the original hyperspectral signal. To address this challenge, we propose the Frequency-Spatial Reciprocal-View Learning (FSRVL) for unsupervised multi-sensor MSI-HSI fusion. 1) Feature Synchronization: weight-sharing convolutions are employed to process Low-Resolution Hyperspectral Image (LR-HSI) and High-Resolution Multispectral Image (HR-MSI) in the frequency domain jointly, achieving an information correspondence between the two modalities with parameter transfer. 2) Frequency Recalibration: sub-pixel information is assigned to learnable filters to adaptively refine spatial distributions across various materials and promote the reestablishment of their spectral characteristics. The advantages of FSRVL were demonstrated across various simulated and real-world scenarios, with experiments confirming that FSRVL outperforms the baselines.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 10244