GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attention

Lingjun Zhao, Sizhe Wei, James Hays, Lu Gan

Abstract:

3D semantic occupancy prediction is essential for
achieving safe, reliable autonomous driving and robotic navigation. Compared to camera-only perception systems, multimodal pipelines, especially LiDAR-camera fusion methods, can
produce more accurate and fine-grained predictions. Although
voxel-based scene representations are widely used for semantic
occupancy prediction, 3D Gaussians have emerged as a continuous and significantly more compact alternative. In this work,
we propose a multi-modal Gaussian-based semantic occupancy
prediction framework utilizing 3D deformable attention, namely
GaussianFormer3D. We introduce a voxel-to-Gaussian initialization strategy that provides 3D Gaussians with accurate geometry priors from LiDAR data, and design a LiDAR-guided 3D
deformable attention mechanism to refine these Gaussians using
LiDAR-camera fusion features in a lifted 3D space. Extensive
experiments on real-world on-road and off-road autonomous
driving datasets demonstrate that GaussianFormer3D achieves
state-of-the-art prediction performance with reduced memory consumption and improved efficiency. Project website:
https://lunarlab-gatech.github.io/GaussianFormer3D/.