BM2CP: Efficient Collaborative Perception with LiDAR-Camera Modalities

Binyu Zhao, Wei ZHANG, Zhaonian Zou

Keywords: Multi-Agent Perception, Multi-Modal Fusion, Vehicle-to-Everything (V2X) Application

Abstract: Collaborative perception enables agents to share complementary perceptual information with nearby agents. This can significantly benefit the perception performance and alleviate the issues of single-view perception, such as occlusion and sparsity. Most proposed approaches mainly focus on single modality (especially LiDAR), and not fully exploit the superiority of multi-modal perception. We propose an collaborative perception paradigm, BM2CP, which employs LiDAR and camera to achieve efficient multi-modal perception. BM2CP utilizes LiDAR-guided modal fusion, cooperative depth generation and modality-guided intermediate fusion to acquire deep interactions between modalities and agents. Moreover, it is capable to cope with the special case that one of the sensors is unavailable. Extensive experiments validate that it outperforms the state-of-the-art methods with 50X lower communication volumes in real-world autonomous driving scenarios. Our code is available at supplementary materials.

Student First Author: yes

Instructions: I have read the instructions for authors (https://corl2023.org/instructions-for-authors/)