Manipulation via Membranes: High-Resolution and Highly Deformable Tactile Sensing and Control

Miquel Oller, Mireia Planas i Lisbona, Dmitry Berenson, Nima Fazeli

Keywords: manipulation, tactile control, deformable tactile sensors

TL;DR: Our method learns soft tactile sensor membrane deformation dynamics to control a grasped object’s pose and force transmitted to the environment during contact-rich manipulation tasks such as drawing and in-hand pivoting.

Abstract: Collocated tactile sensing is a fundamental enabling technology for dexterous manipulation. However, deformable sensors introduce complex dynamics between the robot, grasped object, and environment that must be considered for fine manipulation. Here, we propose a method to learn soft tactile sensor membrane dynamics that accounts for sensor deformations caused by the physical interaction between the grasped object and environment. Our method combines the perceived 3D geometry of the membrane with proprioceptive reaction wrenches to predict future deformations conditioned on robot action. Grasped object poses are recovered from membrane geometry and reaction wrenches, decoupling interaction dynamics from the tactile observation model. We benchmark our approach on two real-world contact-rich tasks: drawing with a grasped marker and in-hand pivoting. Our results suggest that explicitly modeling membrane dynamics achieves better task performance and generalization to unseen objects than baselines.

Student First Author: yes

Website: https://www.mmintlab.com/manipulation-via-membranes/

Code: https://github.com/MMintLab/manipulation_via_membranes