Incorporating Structured Representations into Pretrained Vision \& Language Models Using Scene Graphs

Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, Amir Globerson

Submission Type: Regular Long Paper

Submission Track: Language Grounding to Vision, Robotics and Beyond

Keywords: Vision and language models; Scene graphs; Visio-linguistic compositionality

TL;DR: We propose to improve pretrained VLMs by designing a specialized architecture and a new training method that utilizes a small set of scene graph annotations from Visual Genome that are richer and reflect structured visual and textual information.

Abstract: Vision and language models (VLMs) have demonstrated remarkable zero-shot (ZS) performance in a variety of tasks. However, recent works have shown that even the best VLMs struggle to capture aspects of compositional scene understanding, such as object attributes, relations, and action states. In contrast, obtaining structured annotations, such as scene graphs (SGs), that could improve these models is time-consuming and costly, and thus cannot be used on a large scale. Here we ask whether small SG datasets can provide sufficient information for enhancing structured understanding of pretrained VLMs. We show that it is indeed possible to improve VLMs when learning from SGs by integrating components that incorporate structured information into both visual and textual representations. For the visual side, we incorporate a special ''SG Component'' in the image transformer trained to predict SG information, while for the textual side, we utilize SGs to generate fine-grained captions that highlight different compositional aspects of the scene. Our method improves the performance of several popular VLMs on multiple VL datasets with only a mild degradation in ZS capabilities.

Submission Number: 18