NormalFlow: Fast, Robust, and Accurate Contact-based Object 6DoF Pose Tracking with Vision-based Tactile Sensors

Hung-Jui Huang, Michael Kaess, Wenzhen Yuan

Keywords: Tactile Sensing, Object Tracking, 3D Reconstruction, In-hand Manipulation

TL;DR: Real-time, robust, and accurate tactile-based tracking for novel objects, including low-textured ones like ping pong balls and eggs. Useful for manipulation, dexterous manipulation, and 3D reconstruction tasks.

Abstract: Tactile sensing is crucial for robots aiming to achieve human-level dexterity. In dexterous manipulation, it plays a critical role in monitoring contact modes and estimating an object's 6DoF pose, both of which are necessary for precise control in multi-fingered hands. In this work, we present NormalFlow, a fast, robust, and real-time tactile-based algorithm that jointly tracks object 6DoF pose and monitors contact. Leveraging the precise surface normal estimation of vision-based tactile sensors, NormalFlow determines object movements by minimizing discrepancies between the tactile-derived surface normals. Our results show that NormalFlow consistently outperforms competitive baselines and can track low-texture objects like flat table surfaces and ping pong balls. Additionally, we present state-of-the-art tactile-based 3D reconstruction results, showcasing the high accuracy of NormalFlow. We believe NormalFlow unlocks new possibilities for high-precision perception and manipulation tasks that involve interacting with objects and tools using hands. The video demo, code, and dataset are available on our website: https://joehjhuang.github.io/normalflow.

Submission Number: 20