High-Fidelity Hyperspectral Snapshot of Physical World: System Architecture, Dataset and Model

Erqi Huang, Maoqi Zhang, Zhan Ma, Linsen Chen, Yiyu Zhuang, Xun Cao

Abstract: It is extremely challenging to acquire high-fidelity video of physical world at high spectral, high spatial and high temporal resolution (H <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{3}$</tex-math></inline-formula> R) granularities simultaneously. Existing hyperspectral scanning cameras offer groundtruth with sufficient spatiospectral resolutions but largely lack temporal details; while recent hyperspectral snapshot cameras (e.g., CASSI, PMVIS) enable high temporal resolution acquisition but present inferior capacity to measure fine-grained spatiospectral components as groundtruth. This work builds a joint snapshot-scanning spectral system (JS <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{4}$</tex-math></inline-formula> ), by means of which the acquired low spatiospectral video-rate snapshots can be guided by synchronous-captured high spatiospectral groundtruth in the hyperspectral propagation. We first register the snapshot-scanning image pairs of the JS <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{4}$</tex-math></inline-formula> using a physical imaging model, and then generate a computational hyperspectral image light dataset (CHILD) having both video-rate snapshot and corresponding scanned groundtruth of a specific scene. Taking advantage of the CHILD, we later develop an end-to-end spectral propagation network (SPN) that applies the spectral guided filter and channel attention mechanism to restore rich, and high-fidelity hyperspectral measurement of the dynamic physical world from limited spatiospectral snapshot. The proposed SPN is evaluated extensively using non-blind, blind and semi-blind experiments. For the proposed CHILD and other abundant datasets, our SPN outperforms the state-of-the-art methods greatly.