Inverse Problems in Computer Vision using Adversarial Imagination Priors

Hsiao-Yu Fish Tung, Katerina Fragkiadaki

Abstract: Given an image, humans effortlessly run the image formation process backwards in their minds: they can tell albedo from shading, foreground from background, and imagine the occluded parts of the scene behind foreground objects. In this work, we propose a weakly supervised inversion machine trained to generate similar imaginations that when rendered using differentiable, graphics-like decoders, produce the original visual input. We constrain the imagination spaces by providing exemplar memory repositories in the form of foreground segmented objects, albedo, shading, background scenes and imposing adversarial losses on the imagination spaces. Our model learns to perform such inversion with weak supervision, without ever having seen paired annotated data, that is, without having seen the image paired with the corresponding ground-truth imaginations. We demonstrate our method by applying it to three Computer Vision tasks: image in-painting, intrinsic decomposition and object segmentation, each task having its own differentiable renderer. Data driven adversarial imagination priors effectively guide inversion, minimize the need for hand designed priors of smoothness or good continuation, or the need for paired annotated data.

TL;DR: We present a model that given a visual image learns to generate imaginations of complete scenes, albedo, shading etc, by using adversarial data driven priors on the imaginations spaces.

Conflicts: cmu.edu, ntu.edu, google.com, berkeley.edu

Keywords: Unsupervised Learning, Deep learning