Every pixel counts++: Joint learning of geometry and motion with 3d holistic understanding

Zhenheng Yang

Abstract: Learning to estimate 3D geometry in a single frame and optical flow from consecutive frames by watching unlabeled videos via deep convolutional network has made significant progress recently. Current state-of-the-art (SoTA) methods treat the two tasks independently. One important assumption of the existing depth estimation methods is that the scenes contain no moving object. In this paper, we propose to address the two tasks as a whole, i.e. to jointly understand per-pixel 3D geometry and motion. This eliminates the need of static scene assumption and enforces the inherent geometrical consistency during the learning process, yielding significantly improved results for both tasks. We call our method as "Every Pixel Counts++" or "EPC++". Various loss terms are formulated to jointly supervise the learning across geometrical cues and effective adaptive training strategy is proposed to achieve better performance