Learning to refine depth for robust stereo estimation

Feiyang Cheng, Xuming He, Hong Zhang

Abstract: Highlights • A multi-branch neural network for disparity estimation is proposed. • Initial disparity map, confidence measures and disparity gradients can be predicted simultaneously. • A continuous MRF is proposed to employ the three channel information to refine initial disparity maps. Abstract Traditional depth estimation from stereo images is usually formulated as a patch-matching problem, which requires post-processing stages to impose smoothness and handle depth discontinuities and occlusions. While recent deep network approaches directly learn a regressor for the entire disparity map, they still suffer from large errors near the depth discontinuities. In this paper, we propose a novel method to refine the disparity maps generated by deep regression networks. Instead of relying on ad hoc post-processing, we learn a unified deep network model that predicts a confidence map and the disparity gradients from the learned feature representation in regression networks. We integrate the initial disparity estimation, the confidence map and the disparity gradients into a continuous Markov Random Field (MRF) for depth refinement, which is capable of representing rich surface structures. Our disparity MRF model can be solved via efficient global optimization in a closed form. We evaluate our approach on both synthetic and real-world datasets, and the results show it achieves the state-of-art performance and produces more structure-preserving disparity maps with smaller errors in the neighborhood of depth boundaries.