Exploring View Sampling Strategy in Novel View Synthesis from Causal Perspectives

Yunze Liu, Zifan Wang, Zhiheng Zhang

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Novel View Synthesis, Causal Reasoning, Sampling Strategy

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Neural Radiance Fields (NeRF) has shown promising performance on synthesize high-quality and realistic images. But it often relies on a large amount of high-quality training data. Instead of extensively sampling training samples to cover various details of scenes, a series of works have studied how to utilize prior knowledge to achieve high-quality novel view synthesis with limited training samples. However, these methods have not explored the essence of this problem, which is how to get the optimal training set under limited view inputs.
ActiveNeRF proposes a method based on an active learning scheme that evaluates the reduction of uncertainty given new inputs, selects samples that provide the maximum information gain, and adds them to the existing training set. Since it is necessary to calculate variance changes, evaluating information gain requires the ground-truth of invisible samples, which is impossible to obtain in real situations. We revisit the view sampling strategies from a causal perspective and achieve efficient sampling without requiring the ground-truth of invisible samples. We also propose a new theoretical framework for the sampling problem in NeRF. We analyze how to obtain the optimal sampling strategy based on our framework. Experiments shows that our conclusion can not only guide sampling, but also can help us design regularization term for general NeRF.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1792