Opinion: Learning Intuitive Physics May Require More Than Visual Data

Ellen Su, Solim LeGris, Todd M. Gureckis, Mengye Ren

Keywords: world models, intuitive physics, cognitive science

TL;DR: Scaling up the size of visual datasets or using a developmentally realistic dataset is insufficient for current deep learning architectures to learn robust representations that support intuitive physics reasoning.

Abstract: Humans expertly navigate the world by building rich internal models founded on an intuitive understanding of physics. Meanwhile, despite training on vast quantities of internet video data, state-of-the-art deep learning models still fall short of human-level performance on intuitive physics benchmarks. This work investigates whether data distribution, rather than volume, is the key to learning these principles. We pretrain a Video Joint Embedding Predictive Architecture (V-JEPA) model on SAYCam, a developmentally realistic, egocentric video dataset partially capturing three children's everyday visual experiences. We find that training on this dataset, which represents 0.01% of the data volume used to train SOTA models, does not lead to significant performance improvements on the IntPhys2 benchmark. Our results suggest that merely training on a developmentally realistic dataset is insufficient for current architectures to learn representations that support intuitive physics. We conclude that varying visual data volume and distribution alone may not be sufficient for building systems with artificial intuitive physics.

Submission Number: 42