Learning to Describe Scenes with Programs

Yunchao Liu, Zheng Wu, Daniel Ritchie, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu

Sep 27, 2018 ICLR 2019 Conference Blind Submission readers: everyone Show Bibtex
  • Abstract: Human scene perception goes beyond recognizing a collection of objects and their pairwise relations. We are able to understand the higher-level, abstract regularities within the scene such as symmetry and repetition. Current vision recognition modules and scene representations fall short in this dimension. In this paper, we present scene programs, representing a scene via a symbolic program for its objects and their attributes. We also propose a model that infers such scene programs by exploiting a hierarchical, object-based scene representation. Experiments demonstrate that our model works well on synthetic data and is able to transfer to real images with such compositional structure. The use of scene programs has enabled a number of applications, such as complex visual analogy-making and scene extrapolation.
  • Keywords: Structured scene representations, program synthesis
  • TL;DR: We present scene programs, a structured scene representation that captures both low-level object appearance and high-level regularity in the scene.
0 Replies