Video-to-Music Generation for Film Production: A Dataset and Framework

Haven Kim, Leduo Chen, Bill Wang, Hao-Wen Dong, Julian McAuley

Keywords: music generation

Abstract: Despite growing interest in video-to-music generation systems, their application in film production remains limited, primarily due to the lack of large-scale datasets containing aligned pairs of movie clips and soundtracks. Although prior work has attempted to construct such a dataset, this comprises only 36.5 hours of data, which is insufficient for training robust models. In this paper, we present CineScore Dataset, a novel dataset comprising pairs of video clips from films and their corresponding soundtracks, curated with a novel methodology that automatically identifies and extracts soundtrack segments from video clips, consisting of 552.70 hours and 76,408 video clips sourced from both public domain movies as well as commercial ones from a publicly available dataset. Our comprehensive objective evaluation results show the usefulness of our dataset for building a soundtrack generation model for film production.

Track: Paper Track

Confirmation: Paper Track: I confirm that I have followed the formatting guideline and anonymized my submission.

Submission Number: 24