BEEF: Building a BridgE from Event to Frame

Jiahang Cao, Mingyuan Sun, Ziqing Wang, Hao Cheng, Qiang Zhang, Renjing Xu

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Event-based Camera, Spiking Neural Network, Object Tracking, Image Recognization

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: A novel-designed event processing framework capable of splitting events stream to frames in an adaptive manner.

Abstract:

Event-based cameras are attracting significant interest as they provide event streams which contain rich edge information with high dynamic range and high temporal resolution. Many state-of-the-art event-based algorithms rely on splitting the events into several fixed groups, which are then aggregated into 2D frames by different event representations. However, the fixed slicing method can result in the omission of crucial temporal information, particularly when dealing with diverse motion scenarios (e.g., high-speed and low-speed). In this work, to build a BridgE from converting Event streams to Frames, we propose BEEF, a novel-designed event processing framework capable of splitting events stream to frames in an adaptive manner. In particular, BEEF integrates a low-energy spiking neural network (SNN) as an event trigger to determine the slicing time based on the spike generation. To guide the SNN in firing spikes at optimal time steps, we introduce the Spiking Position-aware Loss (SPA-Loss) function to modulate the neuron's spiking state. In addition, we develop a novel Feedback-Update training strategy that supervises the SNN to make precise event slicing decisions based on the feedback from the downstream artificial neural network (ANN). The newly sliced dataset by SNN is then used to fine-tune the ANN to improve the overall performance. Extensive experiments demonstrate that our BEEF achieves state-of-the-art performance in event-based object tracking and recognition.
Notably, BEEF provides a brand-new SNN-ANN cooperation paradigm, where the SNN acts as an efficient, low-energy data processor to assist the ANN in improving downstream performance, injecting new perspectives and potential avenues of exploration.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3307