A DNN-Based Refining Method for 3D Point Cloud Reconstructed from Multi-View Images

Takenobu Kiyama, Takamasa Takeda, Hidehiko Shishido, Itaru Kitahara

Abstract: This paper proposes a DNN (Deep Neural Network)-based geometrical error refining method for 3D point cloud reconstructed from multi-view images. Recently, SfM (Structure from Motion), which simultaneously estimates the camera pose and the 3D shape of the target object from multi-view images, has been attracting attention. However, due to several factors such as specular reflection, texture-less surface and occlusion, the geometrical error is occurred frequently. To make the matter not easy, it is difficult to analytically mitigate such errors because the multiple factors are combined each other. In this research, we propose a method that applies DNN to reduce the errors. The network for geometrical error reduction is trained using a 3D point cloud sampled from the surface of the 3D-CG model and a 3D point cloud generated from multi-view images which are given by rendering the 3D-CG model. The network is trained by using a 3D feature extraction suitable for the 3D point cloud to estimate the displacement which is necessary to correct the error. By conducting experiments of our method, the learning is performed so that the loss function reduces the position and omission errors.