CRN: Camera Radar Net for Accurate, Robust, Efficient 3D Perception

Youngseok Kim, Sanmin Kim, Juyeb Shin, Jun Won Choi, Dongsuk Kum

TL;DR: Real-time and LiDAR-level 3D object detection using camera-radar fusion

Abstract: Autonomous driving requires a robust and reliable 3D perception system that includes 3D object detection, tracking, and segmentation. Although recent low-cost camera-based approaches have shown promising results, they are susceptible to poor illumination or bad weather conditions and have a large localization error. Hence, fusing camera with low-cost radar, which provides precise long-range measurement and operates reliably in all environments, is promising but has not yet been thoroughly investigated. In this paper, we propose Camera Radar Net (CRN), a novel camera-radar fusion framework that generates a semantically rich and spatially accurate bird's-eye-view (BEV) feature map for various tasks. To overcome the lack of spatial information in an image, we transform perspective view image features to BEV with the help of sparse but accurate radar points. We further aggregate camera and radar feature maps in BEV using multi-modal deformable attention designed for adaptive fusion given spatially misaligned and ambiguous multi-modal inputs. CRN with a real-time setting operates at 20 FPS while achieving comparable performance to LiDAR detectors on nuScenes, and even outperforms at a 100m perception range. Moreover, CRN with offline setting yields 58.3% NDS, 51.5% mAP at 7 FPS and is ranked first among all camera and camera-radar 3D object detectors. The code will be made publicly available soon.

Track: Original Contribution

Type: PDF

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2304.00670/code)