Towards Ultra High-Speed Hyperspectral Imaging by Integrating Compressive and Neuromorphic Sampling

Mengyue Geng, Lizhi Wang, Lin Zhu, Wei Zhang, Ruiqin Xiong, Yonghong Tian

Abstract:

Hyperspectral and high-speed imaging are both important for scene representation and understanding. However, simultaneously capturing both hyperspectral and high-speed data is still under-explored. In this work, we propose a high-speed
hyperspectral imaging system by integrating compressive sensing sampling with bioinspired neuromorphic sampling. Our
system includes a coded aperture snapshot spectral imager capturing moderate-speed hyperspectral measurement frames and
a spike camera capturing high-speed grayscale dense spike streams. The two cameras provide complementary dual-modality
data for reconstructing high-speed hyperspectral videos (HSV). To effectively synergize the two sampling mechanisms and
obtain high-quality HSV, we propose a unified multi-modal reconstruction framework. The framework consists of a Spike
Spectral Prior Network for spike-based information extraction and prior regularization, coupled with a dual-modality iterative optimization algorithm for reliable reconstruction. We finally build a hardware prototype to verify the effectiveness of
our system and algorithm design. Experiments on both simulated and real data demonstrate the superiority of the proposed
approach, where for the first time to our knowledge, high-speed HSV with 30 spectral bands can be captured at a frame rate
of up to 20,000 FPS.