I Know Kung Fu: Synthetic Dexterous Hand Demonstration Collection via VR Teleoperation

Kara Lu, Yanzi He, Cohen Lu, Peihao Li

Keywords: VR-Teleoperation, Dexterous Manipulation

Abstract: In robotic dexterous manipulation learning, limitations in the acquisition of large and scalable amounts of high-quality demonstration data have been a critical bottleneck. Although recent works show significant advances in increasing efficiency through the use of VR technology, human video, or synthetic demonstration generation techniques, these methods have limitations in accuracy and effectiveness. In this project, we propose a data collection pipeline that uses a VR-teleoperation system to track human hand motions in order to collect dexterous hand demonstrations in simulation. Our method utilizes a wrist motion tracker for hand orientation tracking, followed by a dexterous retargeting module to sync human movement with robot movement in real time. We will also implement demonstration augmentation, ultimately yielding multiple distinct, successful trajectories across different scenarios from a single human demonstration.

Submission Type: Short Research Paper (< 4 Pages)

Submission Number: 50