Token to Token Learning From Videos

Jathushan Rajasegaran, Ilija Radosavovic, Yossi Gandelsman, Rahul Ravishankar, Christoph Feichtenhofer, Jitendra Malik

Keywords: Self-supervised learning, Next token prediction, Video models

Abstract: We empirically study generative pre-training from videos. Our approach is conceptually simple and inspired by generative pre-training from images, iGPT. To enable scaling to videos, we make several important improvements along the data, architecture, and evaluation axes. Our model, called Toto, is a causal transformer that generates videos autoregressively, one token at a time. We pre-train our model on a diverse set of videos with over 1 trillion visual tokens. Our tokens are quantized patch embeddings, rather than pixels, and we use relative embeddings for coarse-to-fine pre-training. We conduct a large-scale study across a suite of diverse benchmarks, including image recognition, video classification, object tracking, robotic manipulation and scaling behaviours. We find that, despite minimal inductive biases, our approach achieves competitive performance across all benchmarks.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4394