CMDAR: A Chinese Multi-scene Dynamic Audio Reasoning Benchmark with Diverse Challenges

ACL ARR 2026 January Submission1025 Authors

Keywords: Benchmark, LALMs, Audio reasoning

Abstract: The ability to reason from audio, including speech, environmental sounds, and music, is essential for AI agents to interact effectively in real-world scenarios. Existing benchmarks mainly focus on static or single-scene settings and English audio data and do not fully capture scenarios where multiple speakers, unfolding events, and heterogeneous audio sources interact. To address these challenges, we introduce CMDAR, a chinese benchmark for evaluating models on complex, multi-scene, and dynamically evolving audio reasoning tasks. CMDAR comprises 3,000 carefully curated question–answer pairs linked to diverse audio clips, covering five categories of complex reasoning and spanning three question types. We benchmark 26 state-of-the-art audio language models on CMDAR and observe that they exhibit limitations in complex reasoning tasks. In CMDAR-main, Qwen2.5-Omni (open-source) achieves 76.67\% accuracy, whereas GPT-4o Audio (closed-source) reaches 68.47\%. However, GPT-4o Audio substantially outperforms Qwen2.5-Omni on the more challenging multiple-choice with multiple audios and open-ended tasks. And we provide detail analysis corresponding suggestions for the future development of large audio language models (LALMs). Code and benchmark can be found at {https://anonymous.4open.science/r/CMDAR-C598}.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: corpus creation, benchmarking, NLP datasets, evaluation methodologies, metrics

Contribution Types: Model analysis & interpretability, Data resources, Data analysis

Languages Studied: Chinese, English

Submission Number: 1025