Deceptive-NeRF: Enhancing NeRF Reconstruction using Pseudo-Observations from Diffusion Models

Xinhang Liu, Jiaben Chen, Shiu-hong Kao, Yu-Wing Tai, Chi-Keung Tang

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: NeRF, Few-shot novel view synthesis, diffusion models

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: We introduce Deceptive-NeRF, a novel methodology for few-shot NeRF reconstruction, which leverages diffusion models to synthesize plausible pseudo-observations to improve the reconstruction. This approach unfolds through three key steps: 1) reconstructing a coarse NeRF from sparse input data; 2) utilizing the coarse NeRF to render images and subsequently generating pseudo-observations based on them; 3) training a refined NeRF model utilizing input images augmented with pseudo-observations. We develop a rectification latent diffusion model that adeptly transitions RGB images and depth maps from coarse NeRFs into photo-realistic pseudo-observations, all while preserving scene semantics for reconstruction. Furthermore, we propose a progressive strategy for training the Deceptive-NeRF, using the current NeRF renderings to create pseudo-observations that enhance the next iteration's NeRF. Extensive experiments demonstrate that our approach is capable of synthesizing photo-realistic novel views, even for highly complex scenes with very sparse inputs. Codes will be released.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 480