Points2NeRF: Generating Neural Radiance Fields from 3D point cloud

Dominik Zimny, Tomasz Trzcinski, Przemysław Spurek

Keywords: NeRF, Neural Radiance Fields, 3D point clouds

TL;DR: We convert 3D point clouds into Neural Radiance Fields (NeRFs).

Abstract: Neural Radiance Fields (NeRFs) offer a state-of-the-art quality in synthesising novel views of complex 3D scenes from a small subset of base images. For NeRFs to perform optimally, the registration of base images has to follow certain assumptions, including maintaining constant distance between the camera and the object. We can address this limitation by training NeRFs with 3D point clouds, instead of images, yet a straightforward substitution is impossible due to the sparsity of 3D clouds in the under-sampled regions which leads to incomplete reconstructions output by NeRFs. To solve this problem, here we propose an auto-encoder-based architecture that leverages a hypernetwork paradigm to transfer 3D points with the associated color values through a lower-dimensional latent space and generate weights of NeRF model. This way we are able to accommodate sparsity of 3D point clouds and fully exploit the potential of point cloud data. As a side benefit, our method offers an implicit way for representing 3D scenes and objects, that can be employed to condition NeRFs and hence generalize the models beyond objects seen during training. Empirical evaluation confirms the advantages of our method over conventional NeRFs and proves its superiority in practical applications.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Generative models

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/points2nerf-generating-neural-radiance-fields/code)