View-consistent Object Removal in Radiance Fields

Yiren Lu, Jing Ma, Yu Yin

Abstract: Radiance Fields (RFs) have emerged as a crucial technology for 3D scene representation, enabling the synthesis of novel views with remarkable realism. However, as RFs become more widely used, the need for effective editing techniques that maintain coherence across different perspectives becomes evident. Current methods primarily depend on per-frame 2D image inpainting, which often fails to maintain consistency across views, thus compromising the realism of edited RF scenes. In this work, we introduce a novel RF editing pipeline that significantly enhances consistency by requir- ing the inpainting of only a single reference image. This image is then projected across multiple views using a depth-based approach, effectively reducing the inconsistencies observed with per-frame inpainting. However, projections typically assume photometric consistency across views, which is often impractical in real-world settings. To accommodate realistic variations in lighting and view- point, our pipeline adjusts the appearance of the projected views by generating multiple directional variants of the inpainted image, thereby adapting to different photometric conditions. Additionally, we present an effective and robust multi-view object segmentation approach as a valuable byproduct of our pipeline. Extensive experi- ments demonstrate that our method significantly surpasses existing frameworks in maintaining content consistency across views and enhancing visual quality.

Primary Subject Area: [Generation] Generative Multimedia

Secondary Subject Area: [Experience] Multimedia Applications

Relevance To Conference: Our work utilizes RGB images as inputs to generate a diverse set of outputs including depth maps, edited RGB images, binary masks, and 3D point clouds. These various outputs serve as intermediate representations that span multiple modalities, providing a comprehensive dataset for further processing. Leveraging these data, we designed a novel approach to maintain multi-view consistency through depth projection during Radiance Field inpainting. We also conduct experiments on different modalities of 3D representations such as Neural Radiance Field and 3D Gaussian Splatting. Thus, we think our work is strongly related with multimedia/multimodal. Besides, our work highly align with the Generative Multimedia theme, which aims to generate content with unparalleled realism, and that’s why multi-view consistency so important.

Submission Number: 2516