Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities

Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, Bryan Catanzaro

TL;DR: A parameter-efficient Audio Language Model that achieves sota on several audio understanding and reasoning benchmarks.

Abstract: Understanding and reasoning over non-speech sounds and music are crucial for both humans and AI agents to interact effectively with their environments. In this paper, we introduce Audio Flamingo 2 (AF2), an Audio-Language Model (ALM) with advanced audio understanding and reasoning capabilities. AF2 leverages (i) a custom CLAP model, (ii) synthetic Audio QA data for fine-grained audio reasoning, and (iii) a multi-stage curriculum learning strategy. AF2 achieves state-of-the-art performance with only a 3B parameter small language model, surpassing large open-source and proprietary models across over 20 benchmarks. Next, for the first time, we extend audio understanding to long audio segments (30 secs to 5 mins) and propose LongAudio, a large and novel dataset for training ALMs on long audio captioning and question-answering tasks. Fine-tuning AF2 on LongAudio leads to exceptional performance on our proposed LongAudioBench, an expert-annotated benchmark for evaluating ALMs on long audio understanding capabilities. We conduct extensive ablation studies to confirm the efficacy of our approach.

Lay Summary: People and AI need to understand not only speech but also everyday sounds and music to interact meaningfully with the world. Traditional AI models can handle short snippets of audio but fail to make sense of longer, more complex sound events like a five-minute song or a sequence of machine noises. We created Audio Flamingo 2, a new AI system that learns to “listen” and “reason” about sounds over several minutes by training it on millions of audio clips paired with descriptive text. We improved the audio “ears” (an encoder called AF-CLAP) by feeding it diverse examples, including synthetic questions that teach the system how to answer complex “what” and “when” questions about sounds. We also used a step-by-step training plan that first aligns audio and text, then teaches reasoning on short clips, and finally extends to long audio segments up to five minutes. As a result, Audio Flamingo 2 outperforms larger models on a range of tests, like identifying instruments in music or tracking a conversation in noisy environments. By making AI better at understanding long sounds, our work opens doors to improved assistive devices, smarter monitoring systems, and more engaging audio-based applications.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Link To Code: https://research.nvidia.com/labs/adlr/AF2/

Primary Area: Deep Learning->Large Language Models

Keywords: audio, sound, large audio-language model

Submission Number: 13946