SonicSense: Object Perception from In-Hand Acoustic Vibration

Jiaxun Liu, Boyuan Chen

Keywords: Tactile Perception, Object State Estimation, Audio, Acoustic Vibration Sensing

TL;DR: SonicSense, an integrated hardware and software solution to enable rich object perception capabilities with in-hand acoustic vibration for a multi-finger robot hand.

Abstract: We introduce SonicSense, a holistic design of hardware and software to enable rich robot object perception through in-hand acoustic vibration sensing. While previous studies have shown promising results with acoustic sensing for object perception, current solutions are constrained to a handful of objects with simple geometries and homogeneous materials, single-finger sensing, and mixing training and testing on the same objects. SonicSense enables container inventory status differentiation, heterogeneous material prediction, 3D shape reconstruction, and object re-identification from a diverse set of 83 real-world objects. Our system employs a simple but effective heuristic exploration policy to interact with the objects as well as end-to-end learning-based algorithms to fuse vibration signals to infer object properties. Our framework underscores the significance of in-hand acoustic vibration sensing in advancing robot tactile perception.

Video: https://www.youtube.com/watch?v=MvSYdLMsvx4

Website: http://www.generalroboticslab.com/blogs/blog/2024-06-26-sonicsense/index.html

Code: https://github.com/generalroboticslab/SonicSense?tab=readme-ov-file

Student Paper: yes

Submission Number: 94