PSP-MVSNet: Deep Patch-based Similarity Perceptual Multi-View Stereo

Leiping JIE, Hui Zhang

Abstract: This paper proposes PSP-MVSNet for depth inference problem in multi-view stereo (MVS). We first introduce a novel patch-based similarity perceptual (PSP) module for effectively constructing 3D cost volume. Unlike previous methods that leverage variance-based operators to fuse feature volumes of different views, our method leverages a cosine similarity measure to calculate matching scores for pairs of deep feature vectors and then treats these scores as weights for constructing the 3D cost volume. This is based on an important observation that many performance degradation factors, e.g., illumination changes or occlusions, will lead to pixel differences between multi-view images. We demonstrate that a patch-based cosine similarity can be used as explicit supervision for feature learning and can help speed up convergence. Furthermore, To adaptively set different depth ranges for different pixels, we extend an existing dynamic depth range searching method with a simple yet effective improvement. We can use this improved searching method to train our model in an end-to-end manner and further improve the performance of our method. Experimental results show that our method achieves state-of-the-art performance on the DTU dataset and comparative results on the intermediate set of Tanks and Temples dataset.