Laser: Latent Set Representations for 3D Generative Modeling

Pol Moreno, Adam R. Kosiorek, Heiko Strathmann, Daniel Zoran, Rosalia Galiazzi Schneider, Björn Winckler, Larisa Markeeva, Theophane Weber, Danilo Jimenez Rezende

Keywords: generative models, nerf, computer vision, 3D scenes, novel view synthesis, variational auto-encoder

TL;DR: Generative NeRF with fast inference that can handle large scenes and can inpain unobserved parts of these scenes.

Abstract:

NeRF provides unparalleled fidelity of novel view synthesis---rendering a 3D scene from an arbitrary viewpoint. NeRF requires training on a large number of views that fully cover a scene, which limits its applicability.
While these issues can be addressed by learning a prior over scenes in various forms, previous approaches have been either applied to overly simple scenes or struggling to render unobserved parts.
We introduce Laser-NV---a generative model which achieves high modelling capacity, and which is based on a set-valued latent representation modelled by normalizing flows.
Similarly to previous amortized approaches, Laser-NV learns structure from multiple scenes and is capable of fast, feed-forward inference from few views.
To encourage higher rendering fidelity and consistency with observed views, Laser-NV further incorporates a geometry-informed attention mechanism over the observed views.
Laser-NV further produces diverse and plausible completions of occluded parts of a scene while remaining consistent with observations.
Laser-NV shows state-of-the-art novel-view synthesis quality when evaluated on ShapeNet and on a novel simulated City dataset, which features high uncertainty in the unobserved regions of the scene.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Generative models

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2301.05747/code)