Real-Time Simulation of Deformable Tactile Sensors in Robotic Grasping using Graph Neural Networks

Guillaume Duret, Danylo Mazurak, Frederik heller, Florence Zara, Jan Peters, Liming Chen

Keywords: GNN, Physics simualtion, Tacile sensor, Real time

TL;DR: Real-Time Simulation of Deformable Tactile Sensors in Robotic Grasping using Graph Neural Networks

Abstract:

Real-Time Simulation of Deformable Tactile Sensors in Robotic Grasping using Graph Neural NetworksPhysical simulation plays a crucial role in the development of robotic manipulation methods, and its importance is even greater when dealing with visual tactile sensors in contact-rich scenarios. Despite years of research, simulating such sensors remains highly challenging—both in terms of the underlying physical dynamics and the rendering of tactile images. In this work, we focus exclusively on the physical simulation aspect, leaving the rendering problem outside the scope of our study.

Related work on visual tactile sensor simulation can be broadly divided into two categories: (i) rigid-body simulations and (ii) soft-body simulations. Soft-body approaches offer higher realism by capturing shear forces and deformations under contact with external objects. However, they are significantly more computationally expensive and orders of magnitude slower than rigid-body simulations. In contrast, rigid-body simulations prioritize execution speed, making them suitable for scenarios requiring large-scale data generation, such as reinforcement learning.

This work addresses the speed limitations of soft-body simulations by leveraging Graph Neural Networks (GNNs), which have been successfully applied to learning the physics of deformable objects. We explore the use of GNN models for simulating grasping interactions with visual tactile sensors, achieving performance gains between 100 and 1000 times faster than traditional FEM simulations, while predicting both deformation and stress on the sensor.

Submission Number: 24