FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Driving

Tengju Ye, Wei Jing, Chunyong Hu, Shikun Huang, lingping Gao, Jingke Wang, Fangzhen Li, Weibo Mao, Wencong Xiao, Kun Li, Hang Zheng, Junbo Chen, Kaicheng Yu

Keywords: autonomous driving, planning, prediction, perception, bird's eye view, end-to-end

TL;DR: We propose the first end-to-end autonomous driving multi-tasks framework that support fusion of multi-modality inputs.

Abstract:

Building a multi-modality multi-task neural network toward accurate and robust
performance is a de-facto standard in perception task of autonomous driving.
However, leveraging such data from multiple sensors to jointly optimize the pre-
diction and planning tasks remains largely unexplored. In this paper, we present
FusionAD, to the best of our knowledge, the first unified framework that fuse the
information from two most critical sensors, camera and LiDAR, goes beyond per-
ception task. Concretely, we first build a transformer based multi-modality fusion
network to effectively produce fusion based features. In constrast to camera-based
end-to-end method UniAD, we then establish a fusion aided modality-aware pre-
diction and status-aware planning modules, dubbed FMSPnP that take advantages
of multi-modality features. We conduct extensive experiments on commonly used
benchmark nuScenes dataset, our FusionAD achieves state-of-the-art performance
and surpassing baselines on average 15% on perception tasks like detection and
tracking, 10% on occupancy prediction accuracy, reducing prediction error from
0.708 to 0.389 in ADE score and reduces the collision rate from 0.31% to only
0.12%.

Primary Area: applications to robotics, autonomy, planning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2295