Dynamic Reflections: Probing Video Representations with Text Alignment

Maks Ovsjanikov, Viorica Patraucean, Leonidas Guibas, Tyler Zhu, Tengda Han

Keywords: Platonic Representation hypothesis, video understanding, video-text alignment

TL;DR: Our study of video-text representation alignment demonstrates that alignment is dramatically improved by using richer test-time data, such as multiple video frames and diverse captions.

Abstract:

The alignment of representations from different modalities has recently been shown to provide insights on the structural similarities and downstream capabilities of different encoders across diverse data types.
While significant progress has been made in aligning images with text, the temporal nature of _video_ data remains largely unexplored in this context.
In this work, we conduct the first comprehensive study of video-text representation alignment, probing the capabilities of modern video and language encoders.
Our findings reveal several key insights.
First, we demonstrate that cross-modal alignment highly depends on the richness of both visual (static images vs. multi-frame videos) and text (single caption vs. a collection) data _provided at test time_, especially when using state-of-the-art video encoders.
We propose parametric test-time scaling laws that capture this behavior and show remarkable predictive power against empirical observations.
Secondly, we investigate the correlation between semantic alignment and performance on both semantic and non-semantic downstream tasks, providing initial evidence that strong alignment against text encoders may be linked to _general-purpose_ video representation and understanding.
Finally, we correlate temporal reasoning with cross-modal alignment providing a challenging test-bed for vision and language models.
Overall, our work introduces video-text alignment as an informative zero-shot way to probe the representation power of different encoders for spatio-temporal data.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 15905