How to evaluate monocular depth estimation?

Siyang Wu, Jack Nugent, Willow Yang, Jia Deng

Keywords: monocular depth estimation, evaluation

Abstract: Monocular depth estimation is an important task with rapid progress, but how to evaluate it remains an open question, as evidenced by a lack of standardization in existing literature and an unhelpfully large selection of evaluation metrics whose trade-offs and behaviors are not well understood. This paper contributes a novel, quantitative analysis of existing metrics in terms of their sensitivity to various types of perturbations of ground truth, emphasizing comparison to human judgment. Our analysis reveals that existing metrics are severely under-sensitive to curvature perturbation such as making flat surfaces wavy. To remedy this, we introduce a new metric based on relative surface normals, along with new depth visualization tools and a principled method to create composite metrics with better human alignment. All code, data, and tools will be open-sourced.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 654