NeuroSORT: A Neuromorphic Accelerator for Spike-based Online and Real-time Tracking

Ziyang Shen, Xiaoxu Xie, Chaoming Fang, Fengshi Tian, Shunli Ma, Jie Yang, Mohamad Sawan

Abstract: The increasing need for real-time computation with low-power consumption is driving the advancement of specialized neuromorphic processors on various applications. Multi object tracking, as one of the most challenging tasks in computer vision, has gained wide attention with many solutions proposed. Nevertheless, they consume huge resources and fail to adapt to edge application scenarios with strict power and resource constraints. In this work, we propose NeuroSORT, a neuromorphic accelerator for spike-based online and real-time object tracking, which leverages spiking neural network (SNN) to solve linear assignment problem and explores the hardware acceleration on tracking algorithms. Experimental results show that the proposed accelerator reaches an accuracy of 99.43% on linear assignment task and 69.641 HOTA score on MOT17 dataset, while consuming 0.257mW energy and 0.17mm2 area. The overall power consumption is reduced by 41.1% compared with SOTA works with equivalent performance.