Latent Transformations for Object View Points Synthesis

Sangpil Kim, Nick Winovich, Hyung-gun Chi, Guang Lin, Karthik Ramani

Abstract:

We propose a fully-convolutional conditional generative model, the latent transformation neural network (LTNN), capable of view synthesis using a light-weight neural network suited for real-time applications. In contrast to existing conditional
generative models which incorporate conditioning information via concatenation, we introduce a dedicated network component, the conditional transformation unit (CTU), designed to learn the latent space transformations corresponding to specified target views. In addition, a consistency loss term is defined to guide the network toward learning the desired latent space mappings, a task-divided decoder is constructed to refine the quality of generated views, and an adaptive discriminator is introduced to improve the adversarial training process. The generality of the proposed methodology is demonstrated on a collection of three diverse tasks: multi-view reconstruction on real hand depth images, view synthesis of real and synthetic faces, and the rotation of rigid objects. The proposed model is shown to exceed state-of-the-art results in each category while simultaneously achieving a reduction in the computational demand required for inference by 30% on average.

Keywords: conditional generative model, deep learning, fully-convolutional network, image attribute modification, multi-view reconstruction, view sythesis

TL;DR: We introduce an effective, general framework for incorporating conditioning information into inference-based generative models.