Real-Time Unsupervised Multi-View Depth Estimation Network For Virtual View Synthesis

Ke Qiu, Song Gu, Shiyi Liu, Yawen Lai, Yangang Cai, Ronggang Wang

Abstract: The existing learning-based multi-view stereo (MVS) approaches achieve impressive results compared with traditional methods. However, most of them rely on ground-truth 3D data as supervision, and the acquisition of high-quality ground truth for various scenes is a challenging problem. In this paper, we propose a novel real-time unsupervised multi-view depth estimation network for virtual view synthesis tasks and take multi-view images as supervision. To improve the completeness and accuracy of the virtual viewpoint, we propose a novel shared occlusion mask to deal with the artifacts caused by occlusion in the reconstructed image, and filter out the unreliable points in the depth map. Besides, we also design a mask-based photometric loss to guide our network to generate more reasonable masks and high-quality depth maps. Experimental results on the IEEE1857.9 virtual viewpoint synthesis dataset demonstrate that our proposed method outperforms other recent MVS methods and achieves more excellent real-time performance.