Unifying Spike Perception and Prediction: A Compact Spike Representation Model using Multi-scale Correlation

Kexiang Feng, Chuanmin Jia, Siwei Ma, Wen Gao

Abstract:

The widespread adoption of bio-inspired cameras has catalyzed the development of spike-based intelligent applications.
Despite its innovative imaging principle allows for functionality in extreme scenarios, the intricate nature of spike signals poses processing challenges to achieve desired performance.
Traditional methods struggles to deliver visual perception and temporal prediction simultaneously, and they lack the flexibility needed for diverse intelligent applications.
To address this problem, we analyze the spatio-temporal correlations between spike information at different temporal scales.
A novel spike processing method is introduced for compact spike representations that utilizes intra-scale correlation for higher predictive accuracy.
Additionally, we propose a multi-scale spatio-temporal aggregation unit (MSTAU) that further leverages inter-scale correlation to achieve efficient perception and precise prediction.
Experimental results show noticeable improvements in scene reconstruction and object classification, with increases of **3.49dB** in scene reconstruction quality and **2.20%** in accuracy, respectively.
Besides, the proposed method accommodate different visual applications via switching analysis models, offering a novel perspective for spike processing.

Primary Subject Area: [Engagement] Summarization, Analytics, and Storytelling

Secondary Subject Area: [Experience] Multimedia Applications

Relevance To Conference:

We propose an innovative spike processing method for visual perception and temporal prediction utilizing compact spike representation with high versatility, providing a fresh perspective on spike intelligence.
Taking multi-scale correlation into consideration, our method effectively models spatio-temporal information at different scales, facilitating proactive decision-making in scenarios with diverse semantic complexities.
Experimental results confirm that our method significantly enhance predictive accuracy of spike perception, propelling the widespread adoption and application of spike cameras and further contributing for a tech-harmonious world.

Submission Number: 4003