S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking

Tao Tang, Lijun Zhou, Pengkun Hao, Zihang He, Kalok Ho, Shuo Gu, Zhihui Hao, Haiyang Sun, Kun Zhan, Peng Jia, XianPeng Lang, Xiaodan Liang

TL;DR: A simple yet strong approach for end-to-end 3D multi-object tracking

Abstract: 3D multiple object tracking (MOT) plays a crucial role in autonomous driving perception. Recent end-to-end query-based trackers simultaneously detect and track objects, which have shown promising potential for the 3D MOT task. However, existing methods are still in the early stages of development and lack systematic improvements, failing to track objects in certain complex scenarios, like occlusions and the small size of target object’s situations. In this paper, we first summarize the current end-to-end 3D MOT framework by decomposing it into three constituent parts: query initialization, query propagation, and query matching. Then we propose corresponding improvements, which lead to a strong yet simple tracker: S2-Track. Specifically, for query initialization, we present 2D-Prompted Query Initialization, which leverages predicted 2D object and depth information to prompt an initial estimate of the object’s 3D location. For query propagation, we introduce an Uncertainty-aware Probabilistic Decoder to capture the uncertainty of complex environment in object prediction with probabilistic attention. For query matching, we propose a Hierarchical Query Denoising strategy to enhance training robustness and convergence. As a result, our S2-Track achieves state-of-the-art performance on nuScenes benchmark, i.e., 66.3% AMOTA on test split, surpassing the previous best end-to-end solution by a significant margin of 8.9% AMOTA. We achieve 1st place on the nuScenes tracking task leaderboard.

Lay Summary:

Tracking multiple moving objects in 3D—such as cars, bicycles, and pedestrians—is an important part of how self-driving cars understand the world around them. Recently, new methods have been developed that try to detect and follow these objects all at once, using a single, unified system. These new systems are promising, but they still struggle in challenging situations—like when objects are hidden from view or very small.
In our research, we propose a new method called **S2-Track** that improves this type of all-in-one tracking system to make it more accurate and reliable in real-world driving situations. Our approach focuses on three key areas: starting the tracking more precisely, better handling of uncertain or confusing situations, and making the training process more effective.
Thanks to these changes, our system sets a new record on a widely used self-driving car benchmark and ranks 1st on its tracking leaderboard.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Primary Area: Applications->Computer Vision

Keywords: 3D Multi-Object Tracking, S2-Track

Submission Number: 587