NeuralTouch: Leveraging Implicit Neural Descriptor for Precise Sim-to-Real Tactile Robot Control

Yijiong Lin, Bowen Deng, chenghua lu, Max Yang, John Lloyd, Efi Psomopoulou, Nathan F. Lepora

Keywords: Visual and Tactile Sensing, Dexterous Manipulation, Sim2Real, Neural Fields, Deep Reinforcement Learning

TL;DR: A multi-modal learning framework for generalizable and accurate dexterous manipulation

Abstract:

Grasping accuracy is a critical prerequisite for precise object manipulation, often requiring careful alignment between the robot hand and object. Neural Descriptor Fields (NDF) offer a promising vision-based method to generate grasping poses that generalize across object categories. However, NDF alone can produce inaccurate poses due to imperfect camera calibration, incomplete point clouds, and object variability. Meanwhile, tactile sensing enables more precise contact, but existing approaches typically learn policies limited to simple, predefined contact geometries.
In this work, we introduce NeuralTouch, a multimodal framework that integrates NDF and tactile sensing to enable accurate, generalizable grasping through gentle physical interaction. Our approach leverages NDF to implicitly represent the target contact geometry, from which a deep reinforcement learning (RL) policy is trained to refine the grasp using tactile feedback. This policy is conditioned on the neural descriptors and does not require explicit specification of contact types.
We validate NeuralTouch through ablation studies in simulation and zero-shot transfer to real-world manipulation tasks—such as peg-out/in-hole and bottle lid opening—without additional fine-tuning. Results show that NeuralTouch significantly improves grasping accuracy and robustness over baseline methods, offering a general framework for precise, contact-rich robotic manipulation.

Submission Number: 3