What Happens Next? Anticipating Future Motion by Generating Point Trajectories

Gabrijel Boduljak, Laurynas Karazija, Iro Laina, Christian Rupprecht, Andrea Vedaldi

Keywords: motion generation, point trajectories, flow matching

TL;DR: We develop a video-like diffusion model that generates (quasi-)dense motion trajectories rather than pixels, achieving significantly more accurate and efficient motion generation than prior motion forecasters and video generators.

Abstract: We consider the problem of forecasting motion from a single image, i.e., predicting how objects in the world are likely to move, without the ability to observe other parameters such as the object velocities or the forces applied to them. We formulate this task as conditional generation of dense trajectory grids with a model that closely follows the architecture of modern video generators but outputs motion trajectories instead of pixels. This approach captures scene-wide dynamics and uncertainty, yielding more accurate and diverse predictions than prior regressors and generators. Although recent state-of-the-art video generators are often regarded as world models, we show that they struggle with forecasting motion from a single image, even in simple physical scenarios such as falling blocks or mechanical object interactions, despite fine-tuning on such data. We show that this limitation arises from the overhead of generating pixels rather than directly modeling motion.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 9975