Tactile-Driven Dexterous In-Hand Writing via Extrinsic Contact Sensing

CanZhao, Lingzi Xie, Bidan huang, Shuai Wang, Daolin Ma

Keywords: In-Hand manipulation, force and tactile sensing, reinforcement learning

TL;DR: This work highlights the potential of tactile-driven in-hand manipulation with extrinsic contact, paving the way for more dexterous robotic applications in unstructured environments.

Abstract: Dexterous in-hand manipulation, especially involv- ing interactions between grasped objects and external environ- ments, remains a formidable challenge in robotics. This study tackles the complexities of in-hand manipulation under extrinsic contact through a representative three-finger handwriting task. We propose a hybrid arm-hand coordination framework that combines reinforcement learning with compliance control, offer- ing both flexibility and robustness. Leveraging tactile sensors embedded in each finger, our tactile-driven estimation model dynamically predicts in-hand object pose and external contact, eliminating the need for fixed contact states. The proposed frame- work is first validated in simulation, where it successfully executes diverse writing tasks with accurate contact sensing. Sim-to-Real transfer is achieved through systematic calibration of finger joints and tactile sensors, supported by domain randomization. Real- world experiments further demonstrate the system’s adaptability to writing tools with varying physical properties—such as radius, length, mass, and friction—while maintaining stability across dif- ferent trajectories. Also see https://inhandwriting.github.io/. This work advances robotic manipulation capabilities in unstructured environments.

Submission Number: 2