LE-MVSNet: Lightweight Efficient Multi-view Stereo Network

Changfei Kong, Ziyi Zhang, Jiafa Mao, Sixian Chan, Weigou Sheng

Abstract: Multi-view Stereo(MVS) has been studied for decades as a critical algorithm for 3D reconstruction. Lately, many learning-based methods have improved the reconstruction performance of traditional algorithms, but they pay limited attention to memory consumption and runtime. To address this issue, we propose a novel and effective learning-based MVS framework(LE-MVSNet), based on our exploration of the depth hypothesis and cost volume in this work. Firstly, to decrease the number of depth hypotheses, we establish a more reasonable depth hypothesis space based on its sparse point cloud corresponding to the image set, replacing the previous method of randomly depth hypothesis in evenly divided depth layers within a predefined depth range. Secondly, to reduce memory consumption, we design a lightweight group-wise correlation by compressing the channel of the aggregated cost volumes to one. In addition, for acceleration, we propose SE-UNet, which executes U-Net regularization in the width and height direction, and SE-Net for self-attention in the depth direction. Finally, our method achieves competitive performance on DTU and BlendedMVS dataset with significantly higher efficiency. Compared to MVSNet, our method reduces memory consumption by 52.78\(\%\) and runtime by 88.57\(\%\).