Long-Tailed 3D Detection via 2D Late Fusion

Yechi Ma, Neehar Peri, Shuoquan Wei, Yanan Li, Wei Hua, Deva Ramanan, Shu Kong

Keywords: Long-Tailed 3D Detection, Autonomous Vehicles, Multi-modal Fusion, LiDAR, Object Detection

TL;DR: We present a late-fusion approach that uses 2D RGB detections to filter out 3D LiDAR-based detections to improve Long-Tailed 3D Detection.

Abstract: Autonomous vehicles (AVs) must accurately detect objects from both common and rare classes for safe navigation, motivating the problem of Long-Tailed 3D Object Detection (LT3D). Contemporary LiDAR-based 3D detectors perform poorly on rare classes (e.g., CenterPoint achieves only 5.1 AP on stroller) because it is difficult to recognize objects from sparse LiDAR points alone. RGB images may help resolve such ambiguities, motivating the study of multi-modal RGB-LiDAR fusion. Specifically, we delve into a simple late-fusion framework that ensembles 2D RGB and 3D LiDAR detections and find that (a) high-resolution RGB images help recognize rare objects, (b) LiDAR provides precise 3D localization, and (c) uni-modal detectors can easily leverage more training data because they do not require aligning and annotating multi-modal data. We examine three critical components in this late-fusion framekwork: (1) whether to train 2D or 3D RGB detectors, (2) whether to match RGB and LiDAR detections in 3D or the projected 2D image plane (3) how to fuse matched detections. Extensive experiments reveal that using 2D RGB detectors, matching in the 2D image plane, and fusing scores probabilistically with calibration leads to the state-of-the-art LT3D performance, achieving 51.4 mAP on the established nuScenes LT3D benchmark, improving over prior work by 5.9 mAP.

Primary Area: applications to robotics, autonomy, planning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 88