Fast Post-training Analysis of NeRFs Using A Simple Visibility Prediction Network
Primary Area: representation learning for computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: NeRF, novel view synthesis
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
TL;DR: a fast post-training analysis method for understanding and improving the quality of NeRF rendered images
Abstract: Exercising NeRFs on real-world data taught us that their novel view rendering capability varies across different views and rendering of regions that are visible in more input images often produces more reliable results. However, efficient quantitative tools haven't been developed in this regard to facilitate the post-training analysis of NeRF rendered images. In this paper, we introduce a simple visibility prediction network that efficiently predicts the visibility of \textit{any} point in space from \textit{any} of the input cameras. We further introduce a visibility scoring function that characterizes the reliability of the rendered points, which assists the evaluation of NeRF rendering quality in the absence of ground truth. Utilizing this tool, we also empirically demonstrate two downstream post-training analysis tasks.
The first task is to reduce rendering artifacts via modified volumetric rendering which skips unreliable near-range points. We achieve an average PSNR improvement of 0.6 dB in novel view rendering without changing the network parameters of the pre-trained base NeRF on a benchmark composed of 62 scenes.
The second task is to select additional training images to re-train a NeRF and enhance its rendering quality. By re-training the base NeRF with a handful of additional views selected using the proposed visibility score, we achieve better rendering quality compared to random selection.
Our method is rudimentary, yet efficient and simple to implement making it a suitable drop-in tool for various post-training tasks beyond the studies shown in this paper.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6640
Loading