Interpreting Physics in Video World Models

Sonia Joseph, Quentin Garrido, Randall Balestriero, Matthew Kowal, Thomas Fel, Shahab Bakhtiari, Blake Aaron Richards, Michael Rabbat

Keywords: Applications of interpretability, Feature Geometry, Interpretability for Knowledge Discovery

Other Keywords: video mechanistic interpretability, world model mechanistic interpretability

TL;DR: Mechanistic interpretability of video world models: physics emerges mid-network as a distributed, brain-like circular population code; direction requires high-dim steering (vs. low-dim in LLMs), mirroring V1-to-MT, not physics-engine state.

Abstract:

A long-standing question in physical reasoning is whether video models rely on factorized physical state variables, or on task-specific distributed representations. We present the first mechanistic interpretability study of physical variables inside large-scale video encoders, combining layerwise probing, subspace geometry, patch-level decoding, and targeted attention ablations to characterize where and how physical information is organized.

Across architectures, we identify a sharp intermediate-depth transition, the \emph{Physics Emergence Zone}, at which physical variables become linearly accessible. Scalar speed and acceleration are available from early layers, whereas motion direction emerges only at the Physics Emergence Zone, mirroring the V1 to MT motion hierarchy in primate visual cortex. Direction is encoded as a circular high-dimensional population code: dozens of orthogonal probe dimensions must be steered jointly to change the decoded direction, orders of magnitude more than the low-dimensional steering interventions seen in language models. These findings argue against compact physics-engine state variables and support distributed, hierarchically-organized, ``brain-like'' representations that are nonetheless sufficient for making physical predictions.

Submission Number: 228