Drim-NeRF: Diffusion-Based Restoration for Improving Neural Radiance Fields

Ganlin Yang, Kaidong Zhang, Jingjing Fu, Dong Liu

Abstract: The rendering degradations produced by Neural Radiance Field (NeRF) is a long-standing but complex issue in the field of 3D implicit representation, which arises from a multitude of intricate causes and was not entirely solved by designing complicated scene parameterization methods before. In this paper, we present a diffusion-based restoration method for improving Neural Radiance Field (Drim-NeRF). We consider the NeRF enhancement issue from a low-level restoration perspective by viewing all types of rendering artifacts as a specific degradation model added to clean ground truths. By leveraging the powerful prior knowledge encapsulated in diffusion model, we could restore the high-realism improved renderings conditioned on the raw low-quality rendering counterparts. To further ensure the multi-view consistent rendering enhancement, we innovatively propose to adopt optical flow warping to reduce temporal inconsistency and employ feature-wrapping in VAE decoder to improve fidelity. Our proposed method is easy to implement and agnostic to various NeRF backbones. We conduct extensive experiments on challenging large-scale urban scenes and unbounded 360-degree scenes, as well as other baselines and datasets and achieve substantial qualitative and quantitative improvements, both in the restoration quality and the multi-view consistency perspective.

External IDs:dblp:journals/tcsv/YangZFL26