MimicTouch:Tactile Affordance for Robot Synesthesia for Dexterous Manipulation

ICLR 2026 Conference Submission709 Authors

Keywords: visuotactile affordance

Abstract:

In the field of dexterous robotic manipulation, integrating visual and tactile modalities to inform manipulation policies presents significant challenges, especially
in noncontact scenarios where reliance on tactile perception can be inadequate.
Visual affordance techniques currently offer effective manipulation-centric semantic priors focused on objects. However, most existing research is limited to
using camera sensors and prior object information for affordance prediction. In
this study, we introduce a unified framework called Tactile Affordance in Robot
Synesthesia (TARS) for dexterous manipulation that employs robotic synesthesia
through a unified point cloud representation. This framework harnesses the visuotactile affordance of objects, effectively merging comprehensive visual perception
from external cameras with tactile feedback from local optical tactile sensors to
handle tasks involving both contact and non-contact states. We simulated tactile
perception in a virtual environment and trained task-oriented manipulation policies.
Subsequently, we tested our approach on four distinct manipulation tasks, conducting extensive experiments to evaluate how different modules within our method
optimize the performance of these manipulation policies.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 709