ViFu: Visible Part Fusion for Multiple Scene Radiance Fields

Tianhan Xu, Takuya Ikeda, Koichi Nishiwaki

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Computer vision, 3D reconstruction, Neural radiance fields, Neural rendering

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

In this paper, we propose a method to segment and recover a static, clean background and 360$^{\circ}$ objects from multiple scene observations. Recent works have used neural radiance fields to model 3D scenes and improved the quality of novel view synthesis, while few studies have focused on modeling the invisible or occluded parts of the training images. These under-modeled parts constrain both scene editing and rendering view selection. Our basic idea is that, by observing the same set of objects in various arrangement, so that parts that are invisible in one scene may become visible in others. By fusing the visible parts from each scene, occlusion-free rendering of both background scene and foreground objects can be achieved.

We decompose the multi-scene fusion task into two main components: (1) objects/background segmentation and alignment, where we leverage point cloud-based methods tailored to our novel problem formulation; (2) radiance fields fusion, where we introduce $\textit{visibility field}$ to quantify the visible information of radiance fields, and propose $\textit{visibility-aware rendering}$ for multiple scene fusion, ultimately obtaining clean background and 360$^{\circ}$ object rendering. Comprehensive experiments were conducted on synthetic and real datasets, and the results demonstrate the effectiveness of our method.

The code will be release for research purposes upon paper acceptance.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2692