4D-Editor: Interactive Object-level Editing in Dynamic Neural Radiance Fields via Semantic Distillation

DaDong Jiang, Zhihui Ke, Xiaobo Zhou, Tie Qiu, Xidong Shi, Hao Yan

Keywords: Dynamic NeRF Editing, Computer Vision

TL;DR: An interactive framework for object-level editing in dynamic scenes

Abstract:

This paper targets interactive object-level editing (\textit{e.g.}, deletion, recoloring, transformation, composition) in dynamic scenes.
Recently, some methods aiming for flexible editing static scenes represented by neural radiance field (NeRF) have shown impressive synthesis quality, while similar capabilities in time-variant dynamic scenes remain limited.
To solve this problem, we propose 4D-Editor, an interactive semantic-driven editing framework, allowing editing multiple objects in a dynamic NeRF with user strokes on a single frame.
Specifically, we extend the original dynamic NeRF by incorporating Hybrid Semantic Feature Distillation to maintain spatial-temporal consistency after editing.
In addition, a Recursive Selection Refinement module is presented
to significantly boost object segmentation accuracy within a dynamic NeRF to aid the editing process.
Moreover, we develop Multi-view Reprojection Inpainting to fill holes caused by incomplete scene capture after editing.
Extensive quantitative and qualitative experiments on real application scenarios demonstrate that 4D-Editor achieves photo-realistic editing on dynamic NeRFs.
Project page: \href{https://patrickddj.github.io/4D-Editor}{https://patrickddj.github.io/4D-Editor}

Submission Number: 41