Deep Neural Room Acoustics Primitive

Yuhang He, Anoop Cherian, Gordon Wichern, Andrew Markham

Keywords: audio rendering, neural room impulse response, spatial audio, neural audio rendering

TL;DR: propose a novel framework to learning sound propagation primitive

Abstract: Modeling room acoustics encompasses characterizing the sound propagation dynamics in enclosed 3D spaces and is useful in a variety of settings, including audio-visual simulations, embodied sound source localization, etc. Such dynamics are usually represented using one-dimensional room impulse responses (RIR). However, accurately estimating an RIR is often challenging as sound waves undergo reflections, diffraction, absorption, and scattering along the propagation path. In this paper, we propose a deep learning framework to learn a continuous room acoustic field, dubbed Deep Neural Room Acoustic Primitive (DeepNeRAP), capturing all sound propagation properties in a self-supervised manner; our framework allows the characterization of sound propagation from any source position to any receiver position. Our key idea is to allow two cooperative audio agents to actively probe the 3D space, one emitting and the other receiving sounds at varied positions -- analyzing these emitted and received sounds within our neural framework enables inversely characterizing the room scene acoustically. Our learning formulation is grounded in the physical principles of sound wave propagation, including the properties of globality, reciprocity, superposition, and independence. We present experiments on both synthetic and real-world datasets, demonstrating superior quality of our RIR estimation against closely related methods.

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 481