RealTracker: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos

Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, Christian Rupprecht

Keywords: Point tracking, Optical flow, Motion estimation, Pseudo labelling

TL;DR: We propose a simple point tracking architecture, RealTracker, along with a pseudo-labelling protocol to improve point tracking models. We outperform BootsTAPIR, the state-of-the-art point tracking model, while using 1000x less real data.

Abstract:

Most state-of-the-art point trackers are trained on synthetic data due to the difficulty of annotating real videos for this task.
However, this can result in suboptimal performance due to the statistical gap between synthetic and real videos. In order to understand these issues better, we introduce RealTracker, comprising a new tracking model and a new semi-supervised training recipe. This allows real videos without annotations to be used during training by generating pseudo-labels using off-the-shelf teachers.
The new model eliminates or simplifies components from previous trackers, resulting in a simpler and smaller architecture.
This training scheme is much simpler than prior work and achieves better results using 1,000 times less data.
We further study the scaling behaviour to understand the impact of using more real unsupervised data in point tracking.
The model is available in online and offline variants and reliably tracks visible and occluded points.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3032