The BabyView dataset: High-resolution egocentric videos of infants’ and young children’s everyday experiences

Bria Lorelle Long, Violet Xiang, Stefan Stojanov, Robert Z. Sparks, ZiYin, Grace Keene, Alvin Wei Ming Tan, Steven Y. Feng, Auddithio Nag, Chengxu Zhuang, Virginia A. Marchman, Daniel LK Yamins, Michael Frank

Keywords: egocentric video dataset, self-supervised learning, developmental curriculum, data gap

TL;DR: We present a large developmental egocentric video dataset as an open challenge for robust, human-like AI systems.

Abstract: Human children far exceed modern machine learning algorithms in their sample efficiency, achieving high performance in key domains with much less data than current models. This "data gap'' is a key challenge both for building intelligent artificial systems and for understanding human development. Egocentric video capturing children's experience—their "training data''—is a key ingredient for comparison of humans and models and for the development of algorithmic innovations to bridge this gap. Yet there are few such datasets available, and extant data are low-resolution, have limited metadata, and importantly, represent only a small set of children's experiences. Here, we provide the first release of a large developmental egocentric video dataset—the BabyView dataset—recorded using a high-resolution camera with a large vertical field-of-view and gyroscope/accelerometer data. This 430 hour dataset includes egocentric videos from children spanning 6 months–3 years of age in longitudinal, at-home contexts. We provide gold-standard annotations for the evaluation of speech transcription, speaker diarization, and human pose estimation, and evaluate models in each of these domains. We train self-supervised language and vision models and evaluate their transfer to out-of-distribution tasks including syntactic structure learning, object recognition, depth estimation, and image segmentation. Although performance in each scales with dataset size, overall performance is relatively lower than when models are trained on curated datasets, especially in the visual domain. Our dataset stands as an open challenge for robust, human-like AI systems: how can such systems achieve human-levels of success on the same scale and distribution of training data as humans?

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5797