HarmonicNeRF: Geometry-Informed Synthetic View Augmentation for 3D Scene Reconstruction in Driving Scenarios
Abstract: In the realm of autonomous driving, achieving precise 3D reconstruction of the driving environment is critical for ensuring safety and effective navigation. Neural Radiance Fields (NeRF) have shown promise in creating highly detailed and accurate models of complex environments. However, the application of NeRF in autonomous driving scenarios encounters several challenges, primarily due to the sparsity of viewpoints inherent in camera trajectories and the constraints on data collection in unbounded outdoor scenes, which typically occur along predetermined paths. This limitation not only reduces the available scene information but also poses significant challenges for NeRF training, as the sparse and path-distributed observational data leads to under-representation of the scene's geometry. In this paper, we introduce HarmonicNeRF, a novel approach for outdoor self-supervised monocular scene reconstruction. HarmonicNeRF capitalizes on the strengths of NeRF and enhances surface reconstruction accuracy by augmenting the input space with geometry-informed synthetic views. This is achieved through the application of spherical harmonics to generate novel radiance values, taking into careful consideration the color observations from the limited available real-world views. Additionally, our method incorporates proxy geometry to effectively manage occlusion, generating radiance pseudo-labels that circumvent the limitations of traditional image-warping techniques, which often fail in sparse data conditions typical of autonomous driving environments. Extensive experiments conducted on the KITTI, Argoverse, and NuScenes datasets demonstrate our approach establishes new benchmarks in synthesizing novel depth views and reconstructing scenes, significantly outperforming existing methods. Project page: https://github.com/Jiawei-Yao0812/HarmonicNeRF
Primary Subject Area: [Experience] Multimedia Applications
Relevance To Conference: HarmonicNeRF introduces significant advancements in the field of multimedia processing, particularly in the context of 3D scene reconstruction for autonomous driving. This method enhances existing Neural Radiance Fields (NeRF) by incorporating geometry-informed synthetic view augmentation. By using spherical harmonics, HarmonicNeRF generates novel radiance values, overcoming challenges posed by sparse viewpoints and dynamic environments typical of driving scenarios. This approach not only improves the accuracy and consistency of reconstructed scenes but also demonstrates robustness across various datasets, establishing new benchmarks in synthesizing depth views and scene fidelity. Such enhancements directly contribute to the advancement of autonomous systems and the generalization of machine learning models in camera-based multimedia, making HarmonicNeRF highly relevant to the ACM MULTIMEDIA conference. This relevance is underscored by its application in real-world scenarios, which require handling complex and dynamic multimodal data, a core focus of the conference.
Submission Number: 621
Loading