AVCAPS: AN AUDIO-VISUAL DATASET WITH MODALITY-SPECIFIC CAPTIONS

Parthasaarathy Sudarsanam, Irene Martín-Morató, Aapo Hakala, Tuomas Virtanen

Keywords: Audio-visual dataset, captioning dataset, Multimodal learning

TL;DR: An audio-visual captioning dataset with captions for audio, visual and audio-visual contents separately.

Abstract: In this paper, we introduce AVCaps, an audio-visual captioning dataset that contains separate textual captions for the audio, visual, and audio-visual contents of video clips. The dataset contains 2061 video clips constituting a total of 28.8 hours. We provide up to 5 captions for the audio, visual, and audio-visual content of each clip, crowdsourced separately. Existing datasets focus on a single modality or do not provide modality-specific captions, limiting the study of how each modality contributes to overall comprehension in multimodal settings. Our dataset addresses this critical gap in multimodal research by offering a resource for studying how audio and visual content are captioned individually, as well as how audio-visual content is captioned in relation to these individual modalities. To counter the bias observed in crowdsourced audio-visual captions, which often emphasize visual over audio content, we generated three audio-visual captions for each clip using our crowdsourced captions by leveraging existing large language models (LLMs). We present multimodal and crossmodal captioning and retrieval experiments to illustrate the effectiveness of modality-specific captions in evaluating model performance. Notably, we show that a model trained on LLM-generated audio-visual captions captures audio information more effectively, achieving 14% higher Sentence-BERT similarity on ground truth audio captions compared to a model trained on crowdsourced audio-visual captions. We also discuss the possibilities in multimodal representation learning, question answering, developing new video captioning metrics, and generative AI that this dataset unlocks. The dataset will be freely available online.

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 11288