DiffSound: Differentiable Modal Sound Simulation for Inverse Reasoning

Xutong Jin, Chenxi Xu, Ruohan Gao, Jiajun Wu, Guoping Wang, Sheng Li

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: sound synthesis, differentiable simulation, modal analysis, vibration, audio

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose a differentiable sound simulation framework for physically based modal sound synthesis and inverse problems

Abstract:

Accurately estimating and simulating the physical properties of objects from real-world audio observations is of great practical importance in the field of vision and embodied AI. However, previous differentiable rigid or soft body simulations cannot be directly applied to modal sound synthesis due to the high sampling rate of sound, and previous audio synthesizers do not fully model the physical properties of objects behind the modal analysis.
We propose DiffSound, a differentiable sound simulation framework for physically based modal sound synthesis.
Our framework is capable of solving a range of inverse problems, including object shape, material parameter, and impact position reasoning.
Experimental results demonstrate the effectiveness of our approach, highlighting its ability to accurately estimate physical parameters and reproduce the target sound. Our DiffSound differentiable sound simulator serves as a valuable tool for applications requiring sound synthesis and analysis.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5060