Teleportation, Simulation, or Human Video? Data Utilization Law for Robot Manipulation

ICLR 2026 Conference Submission8842 Authors

Keywords: Data Utility Law, Imitation Learning, Robotic Manipulation

Abstract: Teleoperation, simulation, and human video represent the three primary data sources for robotic manipulation. Teleoperation data offers high quality at a high collection cost, whereas simulation and human video data are cheaper to acquire but introduce significant embodiment gaps. This trade-off has sparked a debate in the robotics community about the most effective data types for training robot policies. To address this, we introduce a data utilization law for robotic manipulation, drawing an analogy from economics to establish a formal "exchange rate" across data types. We quantify data utilization by using a single real-world teleoperated trajectory as a base unit and then measuring the volume of other data (i.e., simulation or human video) required to achieve equivalent performance. Through a comprehensive investigation across three manipulation tasks—training Diffusion Policy and $\pi_{0}$ on over 8000 trajectories—we systematically analyze the interplay between real, simulated, and human video data. Our analysis reveals several key findings: 1) Simulation data generally improves model generalization, with an approximate exchange rate of 8 simulation samples providing the equivalent benefit of 1 teleoperated sample. 2) Human video tends to degrade in-domain model performance, where adding approximately 10 human video samples can negate the benefit of a single teleoperated data point. 3) Whether human video helps generalization or simulation aids in-domain performance varies significantly across tasks. We believe that our work provides crucial insights into balancing the costs of data collection with the computational demands of model training.

Primary Area: applications to robotics, autonomy, planning

Submission Number: 8842