GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction

Xiao Chen, Quanyi Li, Tai Wang, Tianfan Xue, Jiangmiao Pang

Keywords: Active 3D Reconstruction, Next-Best-View Prediction, Generalization, Reinforcement Learning

Abstract: Even with the recent advances in neural radiance rendering (NeRF) enable high-quality digitization of large-scale scenes, the image-capturing process is time-consuming and labor-intensive. Previous works attempt to automate this process using active 3D reconstruction, with the Next-Best-View (NBV) policy being the most well-known. However, the majority of NBV policies are rule-based and only apply to a predefined limited action space, limiting their generalization ability. In this work, we propose \emph{GenNBV}, a novel framework that endows the first free-space NBV policy with generalizability through end-to-end training. This policy is reinforcement learning (RL)-based and empowers a 3D scanning drone to capture from any viewpoint and interact with the environment across diverse scenarios, even those involving unseen structures during training. We also proposed a novel scene representation using action, geometric, and semantic embeddings, to further boost generalizability. To evaluate this NBV policy, we also establish a benchmark using the Isaac Gym simulator with the Houses3K and OmniObject3D datasets. Experiments demonstrate that our approach achieves a 98.26\% and 83.61\% coverage ratio on unseen buildings from these datasets, respectively, outperforming prior solutions.

Primary Area: applications to robotics, autonomy, planning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3523