NOVEL-VIEW ACOUSTIC SYNTHESIS FROM 3D RECONSTRUCTED ROOMS

Jen-Hao Rick Chang, Byeongjoo Ahn, Karren Dai Yang, Brian Hamilton, Jonathan Sheaffer, Anurag Ranjan, Miguel Sarabia, Oncel Tuzel

Abstract: We investigate the benefit of combining blind audio record- ings with 3D scene information for novel-view acoustic syn- thesis. Given audio recordings from 2–4 microphones and the 3D geometry and material of a scene containing multi- ple unknown sound sources, we estimate the sound anywhere in the scene. We identify the main challenges of novel-view acoustic synthesis as sound source localization, separation, and dereverberation. While naively training an end-to-end network fails to produce high-quality results, we show that incorporating room impulse responses (RIRs) derived from 3D reconstructed rooms enables the same network to jointly tackle these tasks. Our method outperforms existing methods designed for the individual tasks, demonstrating its effective- ness at utilizing 3D visual information. In a simulated study on the Matterport3D-NVAS dataset, our model achieves near- perfect accuracy on source localization, a PSNR of 26.44 dB and a SDR of 14.23 dB for source separation and dereverbera- tion, resulting in a PSNR of 25.55 dB and a SDR of 14.20 dB on novel-view acoustic synthesis. Code, pretrained models, and video results are available on the project webpage [1].