High Dynamic Range Imaging for Dynamic Scenes Based on Multi-Level Spike Camera

Zhenkun Zhu, Ruiqin Xiong, Jing Zhao, Rui Zhao, Xiaopeng Fan, Shuyuan Zhu, Tie-Jun Huang

Abstract: Spike camera is a retina-inspired neuromorphic camera which can capture dynamic scenes of high-speed motion by firing a continuous stream of spikes at an extremely high temporal resolution. The limitation in the current design is that each spike only represents the arrival of a fixed amount of photons. It can not deal with strong light areas in which the amount of accumulated photons reaches the pre-specified threshold multiple times within a single readout interval. In this paper, we propose a new spike camera model of high-speed imaging for high dynamic range scenarios. In this scheme, each pixel accumulates the incoming photons persistently and generates a new type of spike stream in which each spike symbol may be associated with different levels, indicating the arrival of different amounts of photons since the last readout. This enables the camera to support dynamic scenes with wider dynamic range. To achieve this, we propose a two-level buffer mechanism, one for photon accumulation and one for spike-firing encoding. We use a register to hold the number of spike-firings which has not been read out yet. At each readout time, the major part in the counter is read out via a carefully designed exponential encoding and the counter is updated. Such encoding and readout strategy enables a very efficient expansion of the dynamic range using a small number of encoding bits. Furthermore, we propose an image reconstruction scheme for the proposed camera, utilizing both spike intervals and spike levels to recover the light intensity. We incorporate Mamba and propose a temporal-spatial selective scan mechanism to extract temporal-spatial correlation within spike streams. We employ a pyramid adaptive filtering and alignment module to achieve coarse-to-fine feature alignment. Experimental results show that the proposed scheme can achieve better imaging quality and outperform the existing spike camera in high dynamic range scenarios.

External IDs:dblp:journals/tcsv/ZhuXZZFZH25