Vision-Free Object 6D Pose Estimation for In-Hand Manipulation via Multi-Modal Haptic Attention

Chanyoung Ahn, SungwooPark, Donghyun Hwang

Keywords: Kinesthetic Sensing, in-hand manipulation, dexterous manipulation

TL;DR: We propose a vision-free approach that integrates multiple haptic sensing modalities among kinesthetic, contact, and proprioceptive signals, as well as their temporal dynamics.

Abstract: Humans are capable of in-hand manipulation without visual feedback, by inferring its pose through haptic feedback. However, in-hand manipulation with multi-fingered robotic hands remains highly challenging due to severe self-occlusion and limited visual accessibility. To address this problem, we propose a vision-free approach that integrates multiple haptic sensing modalities. Specifically, we develop a haptic attention-based pose estimator that captures correlations among kinesthetic, contact, and proprioceptive signals, as well as their temporal dynamics. Experimental results demonstrate that haptic feedback alone enables reliable pose estimation and that contact-rich sensing substantially improves reorientation performance. Our pose estimator achieves average errors of only 4.94 mm in position and 11.6 degrees in orientation during 300 iterations (10 seconds), underscoring the effectiveness of haptic-driven pose estimation for dexterous manipulation. Videos are available at https://cold-young.github.io/haptic-estimator/

Submission Number: 11