Design Decisions that Matter: Modality, State, and Action Horizon in Imitation Learning

Brendan Chharawala, Joshua Li, Stephie Liu, Colin Bellinger, David Liu, Chang Shu, Yue Hu, Pengcheng Xi

Keywords: Assistive Robotics, Dataset Design, Robot Learning

TL;DR: We show that mixing teleoperation modalities, tuning action horizon, and being selective with inclusion of proprioception has significant impact on downstream robot performance with Generalist Robotic Policies.

Abstract: Advances in generalist robot learning models have been fuelled by large-scale demonstration datasets, yet which data are most effective remains underexplored. In particular, the role of teleoperation modality in shaping demonstration quality and downstream learning performance is still poorly understood. In this work, we present a comparative study of two common teleoperation interfaces, VR controllers and a haptic device, for collecting robot demonstrations on a robot platform. We focus on two assistive manipulation tasks, surface wiping and lamp switching, and collect a dataset of 400 human demonstrations. To capture operator workload, each session is evaluated using the NASA Task Load Index. We fine-tune the Octo model on these datasets, systematically varying the inclusion of robot state information and action horizon length. Our results highlight clear differences in data quality across modalities and their downstream impact on imitation learning performance. This study contributes insights into what makes robot learning data ``good" and provides guidance on data collection design for assistive manipulation.

Submission Number: 36