MultiVENT: Multilingual Videos of Events and Aligned Natural Text

NeurIPS 2023 Track Datasets and Benchmarks Submission361 Authors

Keywords: video retrieval, information retrieval, multimodal, dataset, multilingual, video, ocr

TL;DR: We introduce MultiVENT: A dataset of multilingual, event-centric videos grounded in natural language text documents covering five target languages.

Abstract: Everyday news coverage has shifted from traditional broadcasts towards a wide range of presentation formats such as first-hand, unedited video footage. Datasets that reflect the diverse array of multimodal, multilingual news sources available online could be used to teach models to benefit from this shift, but existing news video datasets focus on traditional news broadcasts produced for English-speaking audiences. We address this limitation by constructing MultiVENT, a dataset of multilingual, event-centric videos grounded in text documents across five target languages. MultiVENT includes both news broadcast videos and non-professional event footage, which we use to analyze the state of online news videos and how they can be leveraged to build robust, factually accurate models. Finally, we provide a model for complex, multilingual video retrieval to serve as a baseline for information retrieval using MultiVENT.

Submission Number: 361