MeshMVS: Multi-view Stereo Guided Mesh Reconstruction

Rakesh Shrestha, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Qingkun Su, Ping Tan

Keywords: Mesh Reconstruction, Multi-view Stereo, Deep Learning

Abstract:

Deep learning based 3D shape generation methods generally utilize latent features extracted from color images to encode the objects' semantics and guide the shape generation process. These color image semantics only implicitly encode 3D information, potentially limiting the accuracy of the generated shapes. In this paper we propose a multi-view mesh generation method which incorporates geometry information in the color images explicitly by using the features from intermediate 2.5D depth representations of the input images and regularizing the 3D shapes against these depth images. Our system first predicts a coarse 3D volume from the color images by probabilistically merging voxel occupancy grids from individual views. Depth images corresponding to the multi-view color images are predicted which along with the rendered depth images of the coarse shape are used as a contrastive input whose features guide the refinement of the coarse shape through a series of graph convolution networks. Attention-based multi-view feature pooling is proposed to fuse the contrastive depth features from different viewpoints which are fed to the graph convolution networks.
We validate the proposed multi-view mesh generation method on ShapeNet, where we obtain a significant improvement with 34% decrease in chamfer distance to ground truth and 14% increase in the F1-score compared with the state-of-the-art multi-view shape generation method.

One-sentence Summary: We propose multi-view stereo guided mesh reconstruction method which incorporates geometry information from depth images, along with the rendered depth images to refine the mesh in a coarse-to-fine manner.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2010.08682/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=9EA_E7zsn