Towards Reconstruction of 3D Shapes in a Realistic Environment

Mohammad Zohaib, Matteo Taiana, Alessio Del Bue

Abstract: This paper presents an end-to-end approach for single-view 3D object reconstruction in a realistic environment. Most of the existing reconstruction approaches are trained on synthetic data and they fail when evaluated on real images. On the other hand, some of the methods require pre-processing in order to separate an object from the background. In contrast, the proposed approach learns to compute stable features for an object by reducing the influence of image background. This is achieved by feeding two images simultaneously to the model; synthetic with white background and its realistic variant with a natural background. The encoder extracts the common features from both images and hence separates features of the object from features of the background. The extracted features allow the model to predict an accurate 3D object surface from a real image. The approach is evaluated for both real images of the Pix3D dataset and realistic images rendered from the ShapeNet dataset. The results are compared with state-of-the-art approaches in order to highlight the significance of the proposed approach. Our approach achieves an increase in reconstruction accuracy of approximately 6.1% points in F $$_1$$ score with respect to Mesh R-CNN on the Pix3D dataset.