Towards Diverse and Efficient Audio Captioning via Diffusion Models

Manjie Xu, Chenxing Li, Yong Ren, Xinyi Tu, Ruibo Fu, Wei Liang, Dong Yu

Abstract: We introduce Diffusion-based Audio Captioning (DAC), a non-autoregressive diffusion model tailored for diverse and efficient audio captioning. Although existing captioning models relying on language backbones have achieved remarkable success in various captioning tasks, their insufficient performance in terms of generation speed and diversity impedes progress in audio understanding and multimedia applications. Our diffusion-based framework offers unique advantages stemming from its inherent stochasticity and holistic context modeling in captioning. Through rigorous evaluation, we demonstrate that DAC not only achieves superior performance levels compared to existing benchmarks in the caption quality, but also significantly outperforms them in terms of generation speed and diversity.

External IDs:dblp:conf/interspeech/XuLRTFLY25