Weakly Supervised Scene Graph Grounding

Yizhou Zhang, Zhaoheng Zheng, Yan Liu

Keywords: Weakly Supervised Learning, Scene Graph Grounding, Visual Relation, Computer Vision

Abstract: Recent researches have achieved substantial advances in learning structured representations from images. However, current methods rely heavily on the annotated mapping between the nodes of scene graphs and object bounding boxes inside images. Here, we explore the problem of learning the mapping between scene graph nodes and visual objects under weak supervision. Our proposed method learns a metric among visual objects and scene graph nodes by incorporating information from both object features and relational features. Extensive experiments on Visual Genome (VG) and Visual Relation Detection (VRD) datasets verify that our model post an improvement on scene graph grounding task over current state-of-the-art approaches. Further experiments on scene graph parsing task verify the grounding found by our model can reinforce the performance of the existing method.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We propose the task of weakly supervised scene graph grounding and provide a state-of-the-art solution.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=xZ82dD_7Ys