TactileLab: Efficient Shear-Sensitive Tactile Simulation for Sim-to-Real Multi-Finger Dexterous Manipulation

Yijiong Lin, Nathan F. Lepora

Keywords: Tactile Sensing, Deep Reinforcement Learning, Dexterous Manipulation, Sim2Real

TL;DR: TactileLab is a GPU-parallel tactile simulation and learning framework that adds shear-sensitive tactile feedback to enable stronger multi-finger dexterous manipulation.

Abstract: Tactile sensing is especially important for dexterous manipulation with multi-finger hands, where control depends on both normal contact geometry and tangential interaction. Yet existing scalable tactile simulators remain limited in representing shear-sensitive contact efficiently. We present TactileLab, a unified tactile learning framework in IsaacLab for GPU-parallelized multi-modal tactile simulation, large-scale reinforcement learning, real-to-sim tactile transfer, and sim-to-real policy deployment. Our key contribution is a shear-sensitive tactile representation that combines contact depth with a dense tangential displacement field, yielding a compact and transferable tactile observation suitable for efficient rigid-body simulation. While TactileLab supports a broad range of contact-rich tasks, we focus here on multi-finger dexterous in-hand manipulation, a challenging setting with distributed and continuously evolving contacts. In preliminary in-hand object rotation experiments, we further increase difficulty through randomized initial hand orientation and external force/torque perturbations. Under this setting, rich tactile feedback achieves a mean return of 103.5 and a mean episode length of 552.7, improving mean return by 114.3\% over HORA and 32.4\% over AnyRotate, while improving mean episode length by 31.2\% and 8.5\%, respectively. These results indicate that efficient shear-sensitive tactile simulation can provide a strong foundation for learning and transferring dexterous multi-finger manipulation skills.

Submission Number: 18