ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control

Yutong Liang, Quanquan Peng, Ri-Zhao Qiu, Xiaolong Wang

Keywords: Dexterous manipulation, In-hand manipulation, Reference tracking, Multi-objective reinforcement learning

TL;DR: Dexterous hand tracking becomes much easier once the object is kept on track, while the remaining freedom carry the motion style.

Abstract: Human demonstrations provide strong priors for robot manipulation, yet transferring them to real robots is non-trivial because of the kinematic gap. In dexterous manipulation, even simulation tracking remains difficult for long-horizon, contact-rich sequences: a reference-tracking policy must keep objects on their target trajectories while preserving demonstrated joint motion and contact timing. Existing approaches often rely on hand-crafted reward tuning, which requires per-sequence tuning and can break under limited interaction budgets. We introduce ConTrack, a reinforcement learning framework that scales with tracking data. ConTrack treats object tracking as a constraint and allocates the remaining control authority to motion fidelity, allowing the task--style trade-off to adapt online through a dual-variable update. ConTrack also stabilizes long-horizon learning with an adaptive mid-trajectory reset library that reuses policy-reachable simulator states. Qualitative and quantitative results in simulation tracking and on a real robot show that ConTrack improves success and object pose accuracy over prior methods while preserving joint and contact fidelity.

Submission Number: 1