Learning multi-view manifold for single image based modeling

Jiahao Cui, Shuai Li, Qing Xia, Aimin Hao, Hong Qin

Abstract: Highlights • Multi-view GAN can capture data distribution in multi-view manifold space and the whole latent space of 3D objects. • A novel discriminator which is aware of geometry consistency and view diversity across a group of images. • The prominent of learned representations in navigating the manifold of views and shapes to detail a 3D object as well as in generating new shapes. Abstract Image based modeling has an inherent problem that the complete geometry and appearance of a 3D object cannot be directly acquired from limited 2D images, namely reconstruction of a 3D object when only sporadic views are available is challenging due to occlusions and ambiguities within limited views. In this paper, we present a generative network architecture to address the problem of single image based modeling by learning multi-view manifold of 3D objects, which we call Multi-view GAN. Penalties for shape identity consistency and view diversity are introduced to guide the learning process, and Multi-view GAN can provide a powerful representation which consists of 3D descriptors both for shape and view. This disentangled and oriented representation affords us to explore the manifold of views, thus one can detail a 3D object without “blind spot” even if only single view is available. We have evaluated our method on multi-view and 3D shape generation with a wide range of examples, and both qualitative and quantitative results demonstrate that our Multi-view GAN significantly outperforms state-of-the-art methods. Graphical abstract Given a single view image as input, Multi-view GAN produces a unified representation which consists of a 3D shape descriptor and a view descriptor. The complete representation is learned from the manifold of training data, especially from the multi-view manifold. For single image based modeling, the representation is able to synthesize identity preserved images at plentiful viewpoints specified by the view descriptors in embed-ding space of views. Download : Download high-res image (195KB) Download : Download full-size image