Real-Time Simulation of Deformable Tactile Sensors in Robotic Grasping using Graph Neural Networks

Guillaume Duret, Danylo Mazurak, Frederik Heller, Florence Zara, Jan Peters, Liming Chen

Keywords: GNN, Physics simulation, Tactile sensor, Real time

TL;DR: Real-Time Simulation of Deformable Tactile Sensors in Robotic Grasping using Graph Neural Networks

Abstract:

Physical simulation plays a crucial role in the development of robotic manipulation methods, and its importance is even greater when dealing with visual tactile sensors in contact-rich scenarios. Despite years of research, simulating such sensors remains highly challenging—both in terms of the underlying physical dynamics and the rendering of tactile images. In this work, we focus exclusively on the physical simulation aspect, leaving the rendering problem outside the scope of our study.

Related work on visual tactile sensor simulation can be broadly divided into two categories: (i) rigid-body simulations and (ii) soft-body simulations. Rigid-body simulations prioritize execution speed, making them suitable for scenarios requiring large-scale data generation, such as reinforcement learning. In contrast, soft-body approaches offer greater realism by capturing shear forces and deformations under contact with external objects. However, they are significantly more computationally expensive and orders of magnitude slower than rigid-body simulations.

This work addresses the speed limitations of soft-body simulations by leveraging Graph Neural Networks (GNNs). We explore the use of GNN models for simulating grasping interactions with visual tactile sensors, achieving performance gains between $10^2$ and $10^3$ times faster than traditional FEM simulations, while predicting both deformation and stress on the sensor.
The code is available at: https://tacgraspnets.github.io

Submission Number: 24