LeCO-NeRF: Learning Compact Occupancy for Large-scale Neural Radiance Fields

Zhenxing Mi, Dan Xu

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Occupancy, Neural Radiance Fields, Mixture of Experts, Large-scale scene, Novel view synthesis

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: This paper introduces a Heterogeneous Mixture of Experts (HMoE) structure, an imbalanced gate loss and a density loss to efficiently learn compact occupancy representation for large-scale scenes.

Abstract: Neural Radiance Fields (NeRFs) have shown impressive results in modeling large-scale scenes. A critical problem is how to effectively estimate the occupancy to guide empty-space skipping and point sampling. Although grid-based methods show their advantages in occupancy estimation for small-scale scenes, large-scale scenes typically have irregular scene bounds and more complex scene geometry and appearance distributions, which present severe challenges to the grid-based methods for handling large scenes, because of the limitations of predefined bounding boxes and grid resolutions, and high memory usage for grid updating. In this paper, we propose to learn a compact and efficient occupancy representation of large-scale scenes. Our main contribution is to learn and encode the occupancy of a scene into a compact MLP in an efficient and self-supervised manner. We achieve this by three core designs. First, we propose a novel Heterogeneous Mixture of Experts (HMoE) structure with common Scene Experts and a tiny Empty-Space Expert. The heterogeneous structure can be effectively used to model the imbalanced unoccupied and occupied regions in NeRF where unoccupied regions need much fewer parameters. Second, we propose a novel imbalanced gate loss for HMoE, motivated by the prior that most of the 3D points are unoccupied. It enables the gating network of HMoE to accurately dispatch the unoccupied and occupied points. Third, we also design an explicit density loss to guide the gating network. Then, the occupancy of the entire large-scale scene can be encoded into a very compact gating network of the HMoE. As far as we know, we are the first to learn the compact occupancy of large-scale NeRF by an MLP. We show in the experiments that our occupancy network can very quickly learn more accurate, smooth, and clean occupancy compared to the occupancy grid. With our learned occupancy as guidance for empty space skipping, our method can consistently obtain $2.5\times$ speed-up on the state-of-the-art method Switch-NeRF, while achieving highly competitive performances on several challenging large-scale benchmarks.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 802