Overcoming Distribution Shifts with Autonomous Embodied Data Collection

Pengyu Mo, Saimai Lau, Jensen Gao, Dorsa Sadigh, Jennifer Grannen

Keywords: Domain Adaptation, Active Data Collection

TL;DR: We propose using autonomous robots to collect targeted data that bridges distribution gaps in foundation models’ spatial reasoning capabilities, improving depth estimation by 17.5%.

Abstract: Distribution shifts are a fundamental challenge in machine learning that can significantly limit model deployment, especially in robotics where models must handle messy real-world scenarios. The most direct way to overcome these shifts is to collect additional data suited for the deployment domain. However, collecting this data manually can be expensive and difficult to specify. In this work, we propose Autonomous Embodied Data Adaptation (AEDA), where we instead leverage autonomous robotic systems themselves to collect data targeted at overcoming distribution shifts. AEDA uses large multimodal models (LMMs) to identify distribution shifts, construct data collection plans to address them, and execute these plans on a robot. We instantiate AEDA on a real-world mobile manipulator to improve depth estimation of a pretrained foundation model, and show that its data improves prediction accuracy by 17.5% overall compared to a baseline. Additional videos and details: https://robot-data-collector.github.io

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 32