General In-Hand Object Rotation with Vision and Touch

Haozhi Qi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, Jitendra Malik

Keywords: In-Hand Object Rotation, Tactile Sensing, Reinforcement Learning, Transformer, Visuotactile Manipulation

TL;DR: We present a reinforcement learning policy capable of rotating a diverse set of objects over multiple axes using its fingertips.

Abstract: We introduce RotateIt, a system that enables fingertip- based object rotation along multiple axes by leveraging multi- modal sensory inputs. Our system is first trained in simulation, where it has access to ground-truth object shapes and physical properties. Then we distill it to operate solely on realistic yet noisy visual, tactile, and proprioceptive sensory inputs. These multimodal inputs are fused via a visuotactile transformer, enabling online inference of object shapes and physical properties during deployment. Our work highlights that incorporating visual and tactile sensing enables the policy to rotate diverse ob- jects over multiple axes, and significantly outperforms previous methods. Website: https://haozhi.io/rotateit/.

Submission Number: 17