When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network

Dong Xiao, Guangyao Chen, Peixi Peng, Yangru Huang, Yifan Zhao, Yongxing Dai, Yonghong Tian

TL;DR: We propose an asynchronous hybrid anomaly detection network that leverages event data to accurately detect anomalies during driving with extremely low response time.

Abstract: Anomaly detection is essential for the safety and reliability of autonomous driving systems. Current methods often focus on detection accuracy but neglect response time, which is critical in time-sensitive driving scenarios. In this paper, we introduce real-time anomaly detection for autonomous driving, prioritizing both minimal response time and high accuracy. We propose a novel multimodal asynchronous hybrid network that combines event streams from event cameras with image data from RGB cameras. Our network utilizes the high temporal resolution of event cameras through an asynchronous Graph Neural Network and integrates it with spatial features extracted by a CNN from RGB images. This combination effectively captures both the temporal dynamics and spatial details of the driving environment, enabling swift and precise anomaly detection. Extensive experiments on benchmark datasets show that our approach outperforms existing methods in both accuracy and response time, achieving millisecond-level real-time performance.

Lay Summary:

Ensuring the safety of self-driving cars relies on their ability to quickly notice and respond to unexpected events, like a pedestrian stepping into the road or another car making a sudden move.

Our research introduces a new approach that allows autonomous vehicles to detect these anomalies both quickly and accurately. We achieve this by combining two types of data: standard video from regular cameras and rapid, detailed signals from special sensors called event cameras. Event cameras can capture tiny changes in the environment almost instantly, while regular cameras provide a broader view.

By merging these two sources of information in a specially designed system, our method lets self-driving cars recognize dangers in real time, responding within just milliseconds. This advancement brings us closer to safer roads, as self-driving cars become better equipped to handle sudden risks and prevent accidents before they happen.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Link To Code: https://github.com/PKU-XD/EventAD

Primary Area: Applications->Computer Vision

Keywords: Anomaly Detection, Multimodal Network, Autonomous Driving

Submission Number: 12326