Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Aware Contact Representation

Jiahe Pan, Stelian Coros, Jitendra Malik, Toru Lin

Keywords: Dexterous manipulation, Tactile sensing, Sim-to-real

TL;DR: We introduce CoP, a novel tactile representation grounded in physical principles that preserves dense contact information while maintaining robustness for sim-to-real transfer on challenging, contact-rich manipulation tasks.

Abstract: A primary bottleneck in contact-rich manipulation is the difficulty of collecting real-world data. Sim-to-real reinforcement learning offers a scalable alternative, but the simulation-reality gap prevents information-dense modalities like touch from being effectively used. Existing sim-to-real methods mitigate this gap by simplifying tactile data into low-dimensional features -- sacrificing the richness required for complex manipulation. In this work, we introduce CoP, a novel tactile representation grounded in physical principles that preserves dense contact information while maintaining robustness for sim-to-real transfer. To support this representation, we further propose a sensor calibration scheme based on differentiable dynamics, enabling the estimation of taxel orientations without requiring ground-truth force measurements. We evaluate CoP on two challenging contact-rich manipulation tasks: peg-in-hole insertion and ball balancing. Results demonstrate that policies conditioned on CoP achieve successful direct sim-to-real transfer on multi-fingered hands, significantly outperforming both binary and raw tactile baselines. Moreover, the learned policies implicitly capture underlying physical properties, such as object mass, as an emergent byproduct of control.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 24