Diffusion Feature Field for Text-based 3D Editing with Gaussian Splatting

Eunseo Koh, Sangeek Hyun, MinKyu Lee, Jiwoo Chung, Kangmin Seo, Jae-Pil Heo

Keywords: Text-based 3DGS editing, 3DGS generation

TL;DR: We propose DFFSplat, a novel method integrating 3D-consistent diffusion features and a dual-encoder to resolve view inconsistency (Janus problem) in text-based 3D editing, achieving state-of-the-art structural and semantic consistency.

Abstract:

Recent advances in text-based image editing have motivated the extension of these techniques into the 3D domain.
However, existing methods typically apply 2D diffusion models independently to multiple viewpoints, resulting in significant artifacts, most notably the Janus problem, due to inconsistencies across edited views.
To address this, we propose a novel approach termed DFFSplat, which integrates a 3D-consistent diffusion feature field into the editing pipeline.
By rendering and injecting these 3D-consistent structural features into intermediate layers of a 2D diffusion model, our method effectively enforces geometric alignment and semantic coherence across views.
However, averaging 3D features during the feature field learning process can lead to the loss of fine texture details.
To overcome this, we introduce a dual-encoder architecture to disentangle view-independent structural information from view-dependent appearance details.
By encoding only the disentangled structure into the 3D field and injecting it during 2D editing, our method produces semantically and multi-view coherent edited images while maintaining high text fidelity.
Additionally, we employ a time-invariance objective to ensure consistency across diffusion timesteps, enhancing the stability of learned representations.
Experimental results demonstrate that our method achieves state-of-the-art performance in terms of text-fidelity, and better preserves structural and semantic consistency compared to existing approaches.

Primary Area: Applications (e.g., vision, language, speech and audio, Creative AI)

Submission Number: 22065