LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes

shanlin sun, Bingbing Zhuang, Ziyu Jiang, Buyu Liu, Xiaohui Xie, Manmohan Chandraker

Abstract:

Photorealistic simulation plays a crucial role in applications such as autonomous driving, where advances in neural
radiance fields (NeRFs) may allow better scalability through
the automatic creation of digital 3D assets. However, reconstruction quality suffers on street scenes due to largely
collinear camera motions and sparser samplings at higher
speeds. On the other hand, the application often demands
rendering from camera views that deviate from the inputs to
accurately simulate behaviors like lane changes. In this paper, we propose several insights that allow a better utilization
of Lidar data to improve NeRF quality on street scenes. First,
our framework learns a geometric scene representation from
Lidar, which are fused with the implicit grid-based representation for radiance decoding, thereby supplying stronger
geometric information offered by explicit point cloud. Second, we put forth a robust occlusion-aware depth supervision
scheme, which allows utilizing densified Lidar points by accumulation. Third, we generate augmented training views
from Lidar points for further improvement. Our insights
translate to largely improved novel view synthesis under real
driving scenes.