PDTrack: Progressive Dense 3D Tracking

Tuan Duc Ngo, Ashkan Mirzaei, Guocheng Qian, Hanwen Liang, Chuang Gan, Evangelos Kalogerakis, Peter Wonka, Chaoyang Wang

Keywords: 3D tracking

Abstract:

We propose a novel algorithm for accelerating dense long-term 3D point tracking
in videos. Through analysis of existing state-of-the-art methods, we identify two
major computational bottlenecks. First, transformer-based iterative tracking be-
comes expensive when handling a large number of trajectories. To address this,
we introduce a coarse-to-fine strategy that begins tracking with a small subset of
points and progressively expands the set of tracked trajectories. The newly added
trajectories are initialized using a learnable interpolation module, which is trained
end-to-end alongside the tracking network. Second, we adopt a lightweight im-
plementation for the 4D correlation block that reduces its computational cost on
common GPU backends. Together, these improvements lead to a 5–100× speedup
over existing approaches while maintaining state-of-the-art tracking accuracy.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 4541