Learning Context-Adapted Video-Text Retrieval by Attending to User Comments

Laura Hanu, Yuki M Asano, James Thewlis, Christian Rupprecht

Keywords: Multimodal Representation Learning, Video, Text, Retrieval, User Comments

Abstract:

Learning strong representations for multi-modal retrieval is an important problem for many applications, such as recommendation and search. Current benchmarks and even datasets are often manually constructed and consist of mostly clean samples where all modalities are well-correlated with the content. Thus, current video-text retrieval literature largely focuses on video titles or audio transcripts, while ignoring user comments, since users often tend to discuss topics only vaguely related to the video.
In this paper we present a novel method that learns meaningful representations from videos, titles and comments, which are abundant on the internet. Due to the nature of user comments, we introduce an attention-based mechanism that allows the model to disregard text with irrelevant content.
In our experiments, we demonstrate that, by using comments, our method is able to learn better, more contextualised, representations, while also achieving competitive results on standard video-text retrieval benchmarks.

One-sentence Summary: User comments are an overlooked modality that can be leveraged to improve video retrieval