Robust Model-Based In-Hand Manipulation with Integrated Real-Time Motion-Contact Planning and Tracking

Yongpeng Jiang

Keywords: In-hand manipulation, multifingered hands, dexterous manipulation, integrated planning and control

TL;DR: This paper introduces a robust model-based approach for in-hand manipulation with integrated real-time motion-contact planning and tracking. It outperforms existing methods in accuracy, robustness, and real-time performance.

Abstract: Robotic in-hand manipulation, involving fingers making and breaking contacts, advances toward human-like dexterity in real-world robotic interactions. While learning-based approaches have recently shown promising performance, they face bottlenecks due to high data requirements and lengthy training times. Although model-based methods have the potential to overcome these limitations, they struggle with efficient online planning and handling modeling errors, which limits their real-world applications. This paper proposes a novel approach for in-hand manipulation that addresses the limitations of both learning-based and model-based methods. The key feature of our approach is the integrated real-time motion-contact planning and tracking, achieved through a hierarchical structure. At the high level, finger motion and contact force references are jointly generated using contact-implicit model predictive control (MPC). At the low level, these combined references are tracked with tactile feedback. Extensive experiments demonstrate that our approach outperforms existing methods in terms of accuracy, robustness, and real-time performance. It successfully completes all 6 challenging tasks in real-world environments, even under significant external disturbances. The full paper and video are available on https://director-of-g.github.io/in_hand_manipulation_2/.

Submission Number: 12