Motion-Aware Transformer for Multi-Object Tracking

Xu Yang, Gady Agam

Keywords: Object tracking, Multi-object tracking, End-to-end multi-object tracking, Deep Learning, DanceTrack, MOT

Abstract:

Multi-object tracking (MOT) in videos remains challenging due to complex object motions and crowded scenes.
Recent DETR-based frameworks offer end-to-end solutions but typically process detection and tracking queries jointly within a single Transformer Decoder layer,
leading to conflicts and degraded association accuracy.
We introduce the Motion-Aware Transformer (MATR), which explicitly predicts object movements across frames to update track queries in advance.
By reducing query collisions, MATR enables more consistent training and improves both detection and association.
Extensive experiments on DanceTrack, SportsMOT, and BDD100k show that MATR delivers significant gains across standard metrics.
On DanceTrack, MATR improves HOTA by more than 9 points over MOTR without additional data and reaches a new state-of-the-art score of 71.3 with supplementary data.
MATR also achieves state-of-the-art results on SportsMOT (72.2 HOTA) and BDD100k (54.7 mTETA, 41.6 mHOTA) without relying on external datasets.
These results demonstrate that explicitly modeling motion within end-to-end Transformers offers a simple yet highly effective approach to advancing multi-object tracking.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 4590