From Human Hands to Robot Arms: Manipulation Skills Transfer via Trajectory Alignment

Han Zhou, Jinjin Cao, Liyuan Ma, Xueji Fang, Guo-Jun Qi

Keywords: embodied intelligence, vision-language model, human manipulation videos, VLA

TL;DR: Traj2Action bridges the human-robot embodiment gap using 3D trajectories as a unified representation. A coarse trajectory plan guides fine-grained actions via a co-denoising framework, outperforming prior methods in real-world robot tasks.

Abstract:

Learning diverse manipulation skills for real-world robots is severely bottlenecked by the reliance on costly and hard-to-scale teleoperated demonstrations. While human videos offer a scalable alternative, effectively transferring manipulation knowledge is fundamentally hindered by the significant morphological gap between human and robotic embodiments.
To address this challenge and facilitate skill transfer from human to robot, we introduce Traj2Action,a novel framework that bridges this embodiment gap by using the 3D trajectory of the operational endpoint as a unified intermediate representation, and then transfers the manipulation knowledge embedded in this trajectory to the robot's actions.
Our policy first learns to generate a coarse trajectory, which forms an high-level motion plan by leveraging both human and robot data.
This plan then conditions the synthesis of precise, robot-specific actions (e.g., orientation and gripper state) within a co-denoising framework. Extensive real-world experiments on a Franka robot demonstrate that Traj2Action boosts the performance by up to 27% and 22.25% over $\pi_0$ baseline on short- and long-horizon real-world tasks, and achieves significant gains as human data scales in robot policy learning.Our project website, featuring code and video demonstrations, is available at https://anonymous.4open.science/w/Traj2Action-4A45/.

Primary Area: applications to robotics, autonomy, planning

Submission Number: 10037