Joint Spatial-Spectral Pattern Optimization and Hyperspectral Image Reconstruction

Tao Zhang, Zhiyuan Liang, Ying Fu

Abstract: Employing multispectral filter array (MSFA) and several spectral sensitivity functions (SSFs) as hardware encoder and spatial demosaicing (SpaDM) and spectral super-resolution (SpeSR) methods as software decoder to capture hyperspectral image (HSI) is a common method of computational spectral snapshot imaging. Previous works treat SpaDM and SpeSR as two separative processes, and the effect of MSFA and SSF patterns are rarely investigated. In this paper, we propose a deep-learning based method for high quality hyperspectral imaging by joint spatial-spectral optimization, including joint MSFA and SSF optimization, joint SpaDM and SpeSR, and joint pattern optimization and HSI reconstruction. To capture optimal mosaic image in hardware encoder, we design several spatial-spectral pattern optimization layers to automatically optimize the MSFA and SSF. To recover HSI with SpaDM and SpeSR in software decoder, we unfold an optimization algorithm into a convolutional neural network to explicitly consider the observation model and improve the interpretability of the network. Moreover, to boost the quality of hyperspectral imaging, we optimize spatial-spectral pattern and reconstruct HSI in a joint manner. Extensive evaluations and comparisons on HSI spatial and spectral reconstruction demonstrate our joint optimization method outperforms state-of-the-art methods, in terms of both restoration accuracy and visual quality.