Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining
Submission Type: Regular Long Paper
Submission Track: Language Grounding to Vision, Robotics and Beyond
Keywords: vision-and-language, multimodal, pretraining, zero-shot, fine-grained
TL;DR: We propose two approaches to learn from weakly-supervised visual relation data, which improve zero-shot fine-grained understanding in pretrained vision-and-language models.
Abstract: Recent work in vision-and-language pretraining has investigated supervised signals from object detection data to learn better, fine-grained multimodal representations. In this work, we take a step further and explore how we can tap into supervision from small-scale visual relation data. In particular, we propose two pretraining approaches to contextualise visual entities in a multimodal setup. With verbalised scene graphs, we transform visual relation triplets into structured captions, and treat them as additional image descriptions. With masked relation prediction, we further encourage relating entities from image regions with visually masked contexts. When applied to strong baselines pretrained on large amounts of Web data, zero-shot evaluations on both coarse-grained and fine-grained tasks show the efficacy of our methods in learning multimodal representations from weakly-supervised relations data.
Submission Number: 3128
Loading