Two Spectral-Spatial Implicit Neural Representations for Arbitrary-Resolution Hyperspectral Pansharpening

Lin He, Zhou Fang, Jun Li, Jocelyn Chanussot, Antonio Plaza

Abstract: Standard hyperspectral (HS) pansharpening utilizes panchromatic (PAN) images to improve the connected low (spatial) resolution HS (LRHS) images to the spatial resolutions of PANs, while arbitrary-resolution HS (ARHS) pansharpening aims to use PANs to enhance LRHS images to any desired spatial resolutions. For the challenging task of ARHS pansharpening, one of the major obstacles is how to generalize the single pansharpening model learned under predetermined training scales to any pansharpening scales for future data. As implicit neural representations (INRs) have the potential to approximate continuous functions, they offer a possible alternative way to naturally resolve ARHS pansharpening. In this article, we develop two spectral–spatial INRs for ARHS pansharpening: one is a naive spectra–spatial pansharpening INR (NaivePINR) and the other is a dynamic spectra–spatial pansharpening INR (DynamicPINR). The former builds a novel spectral–spatial encoding to produce spectral–spatial priors of observed scenes and uses a spectral–spatial query mapping to reconstruct fine spectral details and spatial details. The latter establishes an innovative twofold tuning mechanism to dynamically adjust both the spectral–spatial encoding and the spectral–spatial query mapping. The experimental results on serval datasets verify the excellent performances of the proposed pansharpening INRs.