Tactile data generation and applications based on visuo-tactile sensors: A review

Yuhao Sun, Ning Cheng, Shixin Zhang, Wenzhuang Li, Lingyue Yang, Shaowei Cui, Huaping Liu, Fuchun Sun, Jianwei Zhang, Di Guo, Wenjuan Han, Bin Fang

Abstract: Tactile sensation is an essential sensory system in humans, providing abilities such as perception and tactile feedback. Due to multisensory information that integrates tactile sensation, humans exhibit remarkable environmental understanding and dexterous manipulation capabilities. Considering the importance of tactile sensing, researchers have developed tactile sensors such as capacitive and piezoresistive types. In recent years, visuo-tactile sensors have won the favor of the research community with abstract tactile information visualization. The Visuo-tactile sensor is an innovative optical sensor that supports image-based tactile information with high resolution compared to electronic tactile sensors, offering new approaches for tactile dataset collection within multimodal datasets. Nevertheless, owing to challenges such as wear resistance, collecting visuo-tactile data remains a high-cost, low-efficiency task, which limits the development of tactile information in multimodal datasets. With the development of the generation methods, visuo-tactile data collection with low efficiency hopes to be solved. Considering the unique contribution of tactile data to multimodal datasets, this review focuses on visuo-tactile data generation. The generation methods for visuo-tactile sensors are categorized into two categories based on simulation approaches: (1) physics-based and (2) learning-based. Additionally, from the perspective of visuo-tactile data, the review summarizes the cutting-edge applications of multimodal datasets incorporating tactile information. Based on this, the challenges and future directions for development are discussed. This review serves as a technical guide for researchers in the field and aims to promote the widespread development and application of multimodal datasets incorporating tactile information.