Event-Enhanced Snapshot Mosaic Hyperspectral Frame Deblurring

Mengyue Geng, Lizhi Wang, Lin Zhu, Wei Zhang, Ruiqin Xiong, Yonghong Tian

Abstract:

Snapshot Mosaic Hyperspectral Cameras (SMHCs)
are popular hyperspectral imaging devices for acquiring both color
and motion details of scenes. However, the narrow-band spectral
filters in SMHCs may negatively impact their motion perception
ability, resulting in blurry SMHC frames. In this paper, we propose
a hardware-software collaborative approach to address the blurring issue of SMHCs. Our approach involves integrating SMHCs
with neuromorphic event cameras for efficient event-enhanced
SMHC frame deblurring. To achieve spectral information recovery
guided by event signals, we formulate a spectral-aware Event-based
Double Integral (sEDI) model that links SMHC frames and events
from a spectral perspective, providing principled model design
insights. Then, we develop a Diffusion-guided Noise Awareness
(DNA) training framework that utilizes diffusion models to learn
noise-aware features and promote model robustness towards camera noise. Furthermore, we design an Event-enhanced Hyperspectral frame Deblurring Network (EvHDNet) based on sEDI, which is
trained with DNA and features improved spatial-spectral learning
and modality interaction for reliable SMHC frame deblurring.
Experiments on both synthetic data and real data show that the
proposed DNA + EvHDNet outperforms state-of-the-art methods
on both spatial and spectral fidelity. The code and dataset will be
made publicly available.