JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments

Zhan Liu, Changli Tang, Yuxin WANG, Zhiyuan Zhu, Youjun Chen, Yiwen Shao, TIANZI WANG, Lei Ke, Zengrui Jin, Chao Zhang

TL;DR: 3D AV-LLM leveraging RGB-D and First-Order Ambisonics for end-to-end grounding and spatial reasoning

Abstract: Current audio-visual large language models (AV-LLMs) are predominantly restricted to 2D perception, relying on RGB video and monaural audio. This design choice introduces a fundamental dimensionality mismatch that precludes reliable source localization and spatial reasoning in complex 3D environments. We address this limitation by presenting JAEGER, a framework that extends AV-LLMs to 3D space, to enable joint spatial grounding and reasoning through the integration of RGB-D observations and multi-channel first-order ambisonics. A core contribution of our work is the neural intensity vector (Neural IV), a learned spatial audio representation that encodes robust directional cues to enhance direction-of-arrival estimation, even in adverse acoustic scenarios with overlapping sources. To facilitate large-scale training and systematic evaluation, we propose SpatialSceneQA, a benchmark of 61k instruction-tuning samples curated from simulated physical environments. Extensive experiments demonstrate that our approach consistently surpasses 2D-centric baselines across diverse spatial perception and reasoning tasks, underscoring the necessity of explicit 3D modelling for advancing AI in physical environments. Our source code, pre-trained model checkpoints, and datasets are available at https://github.com/liuzhan22/JAEGER.

Lay Summary:

Today's AI assistants can watch a video and listen to its audio, but they perceive both as flat, 2D signals — they cannot tell where in a real room a sound is coming from or how far a speaker is sitting. This makes them unreliable for use in physical 3D spaces such as home robots, AR glasses, or assistants for the visually and hearing impaired.

We present JAEGER, an AI model that hears and sees in true 3D. Instead of ordinary video, it takes in depth-aware images that preserve distance, together with a special multi-channel form of audio that captures sound coming from every direction. A learned component we call the Neural Intensity Vector helps the model recover the direction of a sound even when several people talk at once or the room echoes heavily.

To train and evaluate such systems, we also release a large dataset of 61,000 simulated 3D scenes with precise spatial labels. JAEGER consistently outperforms today's 2D audio-visual assistants — a step toward AI that truly understands the physical world.

Link To Code: https://github.com/liuzhan22/JAEGER

Primary Area: Applications->Everything Else

Keywords: AV-LLM, RGB-D, spatial audio, Neural Intensity Vector

Submission Number: 24807