High-Resolution Pixelwise Contact Area and Normal Force Estimation for the GelSight Mini Visuotactile Sensor Using Neural Networks

Niklas Funk, Paul Otto Müller, Boris Belousov, Anton Savchenko, Rolf Findeisen, Jan Peters

Keywords: Visuotactile Sensing, Contact Area Estimation, Normal Force Estimation, Interpretable Contact Representation

TL;DR: We present an approach for pixelwise, high-resolution contact area and normal force estimation using the commercially available GelSight Mini sensor and validate it on force control and marble roll tasks.

Abstract:

Visuotactile sensors are gaining momentum in robotics because they provide high-resolution contact measurements at a fraction of the price of conventional force/torque sensors. It is, however, not straightforward to extract useful signals from their raw camera stream, which captures the deformation
of an elastic surface upon contact. To utilize visuotactile sensors more effectively, powerful approaches are required, capable of extracting meaningful contact-related representations. This paper proposes a neural network architecture called CANFnet that provides a high-resolution pixelwise estimation of the contact area and normal force given the raw sensor images. The CANFnet is trained on a labeled experimental dataset collected using a conventional force/torque sensor, thereby circumventing material identification and complex modeling for label generation. We test CANFnet using GelSight Mini sensors
and showcase its performance on real-time force control and marble rolling tasks. We are also able to report generalization of the CANFnets across different sensors of the same type. Thus, the trained CANFnet provides a plug-and-play solution for pixelwise contact area and normal force estimation for
visuotactile sensors. The models, dataset, and additional information are open-source at https://sites.google.com/view/canfnet.

Confirmation: I am a student and I would like to be considered for the funding award.