Deblurring Neural Radiance Fields with Event-driven Bundle Adjustment

Yunshan Qi, Lin Zhu, Yifan Zhao, Nan Bao, Jia Li

Abstract: Neural Radiance Fields (NeRF) achieves impressive 3D representation learning and novel view synthesis results with high-quality multi-view images as input. However, motion blur in images often occurs in low-light and high-speed motion scenes, which significantly degrades the reconstruction quality of NeRF. Previous deblurring NeRF methods struggle to estimate pose and lighting changes during the exposure time, making them unable to accurately model the motion blur. The bio-inspired event camera measuring intensity changes with high temporal resolution makes up this information deficiency. In this paper, we propose Event-driven Bundle Adjustment for Deblurring Neural Radiance Fields (EBAD-NeRF) to jointly optimize the learnable poses and NeRF parameters by leveraging the hybrid event-RGB data. An intensity-change-metric event loss and a photo-metric blur loss are introduced to strengthen the explicit modeling of camera motion blur. Experiments on both synthetic and real-captured data demonstrate that EBAD-NeRF can obtain accurate camera trajectory during the exposure time and learn a sharper 3D representations compared to prior works.

Primary Subject Area: [Content] Multimodal Fusion

Secondary Subject Area: [Experience] Multimedia Applications, [Generation] Generative Multimedia

Relevance To Conference:

Our research explores the utilization of multimodal data (RGB and event data) to enhance the learning of 3D implicit representations. By incorporating both event data captured by bio-inspired event cameras and image data from traditional RGB cameras, we optimize our network to achieve more robust and accurate results. Through the integration of intensity-change-metric loss and photometric loss, our approach facilitates the joint exploitation of these modalities, enabling the precise modeling of camera motion blur and the generation of clear neural radiance fields.
This multimodal strategy allows us to capture a more comprehensive understanding of dynamic scenes, leveraging the strengths of each data modality to compensate for their respective limitations. In comparison to NeRF and BAD-NeRF methods that rely solely on single-modal RGB data, our approach demonstrates significant improvements in rendering quality and accuracy in motion blur scenes. In summary, our framework not only achieves impressive rendering effects but also opens new avenues for advancing the multimodal 3D scene reconstruction field.

Submission Number: 1058