Camera-Radar Fusion for 3-D Depth Reconstruction

Urs Niesen, Jayakrishnan Unnikrishnan

Abstract: We introduce and study the problem of camera-radar fusion for 3-D depth reconstruction. This problem is motivated by autonomous driving applications, in which we can expect to have access to both front-facing camera and radar sensors. These two sensors are complementary in several respects: the camera is a passive sensor measuring azimuth and elevation; the radar is an active sensor measuring azimuth and range. Fusing their measurements is therefore beneficial. Our fusion solution uses a modified encoder-decoder deep convolutional neural network. We train and evaluate this network on over 100 000 samples collected in highway environments. Our results demonstrate an improvement in reconstruction accuracy and robustness from fusing the two sensors.