Multi-Lingual Acquisition on Multimodal Pre-training for Cross-modal RetrievalDownload PDF

Published: 31 Oct 2022, Last Modified: 06 Jan 2023NeurIPS 2022 AcceptReaders: Everyone
Keywords: cross-lingual cross-modal retrieval, multilingual representation, multimodal representation
Abstract: Vision and diverse languages are important information sources in our living world. A model that understands multi-modalities and multi-languages can be applied to a wider range of real-life scenarios. To build such a multimodal and multilingual model, existing works try to ensemble vision-language data from multiple languages in pre-training. However, due to the large number of languages, these works often require huge computing resources and cannot be flexibly extended to new languages. In this work, we propose a MultiLingual Acquisition (MLA) framework that can easily empower a monolingual Vision-Language Pre-training (VLP) model with multilingual capability. Specifically, we design a lightweight language acquisition encoder based on state-of-the-art monolingual VLP models. We further propose a two-stage training strategy to optimize the language acquisition encoder, namely the Native Language Transfer stage and the Language Exposure stage. With much less multilingual training data and computing resources, our model achieves state-of-the-art performance on multilingual image-text and video-text retrieval benchmarks.
Supplementary Material: zip
TL;DR: This paper proposed MLA, a data-efficient method to empower vision-language pre-training with multilingual capability.
11 Replies