Video-Text Retrieval by Supervised Sparse Multi-Grained Learning
Submission Type: Regular Long Paper
Submission Track: Speech and Multimodality
Submission Track 2: Language Grounding to Vision, Robotics and Beyond
Keywords: Video-Text Retrieval, Multimodal Learning
TL;DR: We introduce S3MA for video-text retrieval, which utilizes a shared sparse space to align video and text and incorporates frame representations for multi-grained alignment with superior performance compared to existing methods.
Abstract: While recent progress in video-text retrieval has been advanced by the exploration of better representation learning, in this paper, we present a novel multi-grained sparse learning framework, S3MA, to learn an aligned sparse space shared between the video and the text for video-text retrieval. The shared sparse space is initialized with a finite number of sparse concepts, each of which refers to a number of words. With the text data at hand, we learn and update the shared sparse space in a supervised manner using the proposed similarity and alignment losses. Moreover, to enable multi-grained alignment, we incorporate frame representations for better modeling the video modality and calculating fine-grained and coarse-grained similarities. Benefiting from the learned shared sparse space and multi-grained similarities, extensive experiments on several video-text retrieval benchmarks demonstrate the superiority of S3MA over existing methods.
Submission Number: 4649
Loading