IMFine: 3D Inpainting via Geometry-guided Multi-view Refinement

Juwei Lu

Abstract:

Current 3D inpainting and object removal methods are
largely limited to front-facing scenes, facing substantial
challenges when applied to diverse, “unconstrained”
scenes where the camera orientation and trajectory are unrestricted.
To bridge this gap, we introduce a novel approach
that produces inpainted 3D scenes with consistent
visual quality and coherent underlying geometry across
both front-facing and unconstrained scenes. Specifically,
we propose a robust 3D inpainting pipeline that incorporates
geometric priors and a multi-view refinement network
trained via test-time adaptation, building on a pretrained
image inpainting model. Additionally, we develop
a novel inpainting mask detection technique to derive targeted
inpainting masks from object masks, boosting the
performance in handling unconstrained scenes. To validate
the efficacy of our approach, we create a challenging and diverse benchmark that spans a wide range of scenes.
Comprehensive experiments demonstrate that our proposed
method substantially outperforms existing state-of-the-art
approaches.