Cross-Sensor Touch Generation

Samanta Rodriguez, Yiming Dou, Miquel Oller, Andrew Owens, Nima Fazeli

Keywords: Tactile Sensing, Cross-Modal Generation, Manipulation, Representation Learning, Soft Bubbles, GelSlims, Digits

TL;DR: We present Touch2Touch and T2D2, two generative models for cross-modal tactile image generation that enable sensor-to-sensor transfer with and without paired data.

Abstract: Today's visuo-tactile sensors come in many shapes and sizes, making it challenging to develop general-purpose tactile representations. This is because most models are tied to a specific sensor design. To address this challenge, we propose two approaches to cross-sensor image generation. The first is an end-to-end method that leverages paired data (Touch2Touch). The second method builds an intermediate depth representation and does not require paired data (T2D2: Touch-to-Depth-to-Touch). Both methods enable the use of sensor-specific models across multiple sensors via the cross-sensor touch generation process. Together, these models offer flexible solutions for sensor translation, depending on data availability and application needs. We demonstrate their effectiveness on downstream tasks such as cup stacking and tool insertion, where models originally designed for one sensor are successfully transferred to another using in-hand pose estimation.

Submission Number: 957