CoVR: Learning Composed Video Retrieval from Web Video Captions

Lucas Ventura, Antoine Yang, Cordelia Schmid, Gül Varol

Keywords: composed image retrieval, deep learning, vision and language, computer vision

TL;DR: This research paper introduces Composed Video Retrieval (CoVR), a task of retrieving relevant videos based on textual descriptions and visual queries, and presents a large dataset for training and evaluating CoVR models.

Abstract:

Composed Image Retrieval (CoIR) has recently gained popularity as a task that considers both text and image queries together, to search for relevant images in a database. Most CoIR approaches require manually annotated datasets, containing image-text-image triplets, where the text describes a modification from the query image to the target image. However, manual curation of CoIR triplets is expensive and prevents scalability. In this work, we instead propose a scalable automatic dataset creation methodology that generates triplets given video-caption pairs. To this end, we mine paired videos with a similar caption from a large database, and leverage a large language model to generate the corresponding modification text. We automatically construct our WebVid-CoVR dataset by applying this procedure to the large WebVid2M collection, resulting in 1.6M triplets. Moreover, we introduce a new benchmark for composed video retrieval (CoVR) and contribute a manually annotated evaluation set, along with baseline results. We further show that training a CoVR model on our dataset transfers well to CoIR, improving the state of the art in the zero-shot setup on both the CIRR and FashionIQ benchmarks. Our code, datasets, and models will be made publicly available

Submission Number: 56