3D Scene De-occlusion in Neural Radiance Fields: A Framework for Obstacle Removal and Realistic Inpainting

Yi LIU, Xinyi LI, Shuai Wenjing

Abstract: Neural Radiance Fields (NeRFs) demonstrate high efficiency in generating photo-realistic novel view. Recent studies introduce the trials on the 3D inpainting by NeRF. However, the performance of these works have been validated for data collected in a narrow range of multi-view, while degrade for the wide range of multi-view. To address this problem, we propose a novel NeRF framework to remove the obstacle and reproduce occluded areas in high quality for both wide and narrow range of multi-view. In this framework, we design a region coding network to carry out object segmentation. With the depth information, the segmentation component transfers a single obstacle mask to other views in high accuracy. By referring to the segmentation results, we introduce an innovative view selection mechanism to reconstruct the occluded area using supplementary information from multi-view and 2D inpainting. We also contribute to the evaluation of 3D scene de-occlusion by introducing a dataset including views captured in wide range and in pair with and without the obstacle object for comparison. We evaluate our framework in both narrow and wide range datasets by quantitative measurement and visually qualitative comparison, which confirm the competitive and superior performance of our framework.

Primary Subject Area: [Experience] Multimedia Applications

Secondary Subject Area: [Experience] Interactions and Quality of Experience

Relevance To Conference: This work proposes a novel NeRF framework to realize 3D de-occlusion and inpainting for rendered scene images. The high quality of the rendered image contributes to the realistic de-occlusion task and makes the proposed work promising in the interactive multimedia applications for VR and AR.

Submission Number: 3938