Endmember-Assisted Camera Response Function Learning, Toward Improving Hyperspectral Image Super-Resolution Performance

Jiangsan Zhao, Ying Qu, Seishi Ninomiya, Wei Guo

Abstract: The camera response function (CRF) that projects hyperspectral radiance to the corresponding RGB images is important for most hyperspectral image super-resolution (HSI-SR) models. In contrast to most studies that focus on improving HSI-SR performance through new architectures, we aim to prevent the model performance drop by learning the CRF of any given HSIs and RGB image from the same scene in an unsupervised manner, independent of the HSI-SR network. Accordingly, we first decompose the given RGB image into endmembers and an abundance map using the Dirichlet autoencoder architecture. Thereafter, a linear CRF learning network is optimized to project the reference HSIs to the RGB image that can be similarly decomposed like the given RGB, assuming that objects in both images share the same endmembers and abundance map. The quality of the RGB images generated from the learned CRFs is compared with that of the corresponding ground-truth images based on the true CRFs of two consumer-level cameras Nikon 700D and Canon 500D. We demonstrate that the effectively learned CRFs can prevent significant performance drop in three popular HSI-SR models on RGB images from different categories of standard datasets of CAVE, ICVL, Chikusei, Cuprite, Salinas, and KSC. The successfully learned CRF using the method proposed in this study would largely promote a wider implementation of HSI-SR models since tremendous performance drop can be prevented practically.