Deep RGB-D Canonical Correlation Analysis For Sparse Depth Completion

Yiqi Zhong, Cho Ying Wu, Suya You, Ulrich Neumann

Abstract: This paper describes our Correlation For Completion Network (CFCNet), an end-to-end deep model to do sparse depth completion based on RGB data. CFCNet utilizes 2D deep canonical correlation analysis as network constraints to ensure that RGB and depth encoders capture the most similar semantics. Then the RGB features are transformed to the depth domain. The complementary RGB information is used to complete the missing depth information. Extensive experiments are performed on both outdoor and indoor scene datasets. For outdoor scenes, KITTI and Cityscape are used, which captured depth information with LiDARs and stereo cameras, respectively. For indoor scenes, NYUv2 with stereo/ORB feature sparsifiers and SLAM RGBD datasets are used. Experiments demonstrate that CFCNet outperforms the state-of-the-art methods using these datasets. Our best results improve the percentage of accurate estimations from 13.03 to 58.89 (+394%) compared with the state-of-the-art method on the SLAM RGBD dataset.

Code Link: https://github.com/choyingw/CFCNet

CMT Num: 2869