DIFFER: Moving Beyond 3D Reconstruction with Differentiable Feature Rendering

Navaneet K. L., Priyanka Mandikal, Varun Jampani, R. Venkatesh Babu

Abstract: Perception of 3D object properties from 2D images form one of the core computer vision problems. In this work, we propose a deep learning system that can simultaneously reason about 3D shape as well as associated properties (such as color, semantic part segments) directly from a single 2D image. We devise a novel depth-aware differentiable feature rendering module (DIFFER) that is used to train our model by using only 2D supervision. Experiments on both synthetic ShapeNet dataset and the real-world Pix3D dataset demonstrate that our 2D supervised DIFFER model performs on par or sometimes even outperforms existing 3D supervised models.