Helping Hand: Completing Collaborative Tasks with Dexterous Hands

Boshi An, Chenyu Yang, Robert K. Katzschmann

Keywords: Dexterous Manipulation, Human-Robot Interaction, Vision-Language-Action Models

TL;DR: Adapt an VLA for real-world dexterous human-robot interaction through body language understanding

Abstract: In this project, we propose a novel approach that enables pre-trained vision-language-action (VLA) models for collaborative dexterous tasks based on body-language. We introduce several methods to improve adaptation to real-world human-robot interaction: leveraging pre-trained vision encoders, incorporating human pose priors, and re-designing the model's action space. Our approach enhances the robot’s ability to perceive, interpret, and respond to human behaviors in a context-aware and data-efficient manner. Real-world evaluations demonstrate the effectiveness of the proposed method.

Submission Number: 6