Structurally Disentangled Feature Fields Distillation for 3D Understanding and Editing

Yoel Levy, David Shavin, Itai Lang, Sagie Benaim

Keywords: Neural Radiance Fields, 3D Segmentation, 3D Editing, Feature Distillation

TL;DR: We distill 2D features into a structurally disentangled feature field that separates view-dependent and view-independent components, leading to improved 3D segmentation and higher-fidelity editing capabilities.

Abstract: Recent work demonstrated the ability to leverage or distill pre-trained 2D features obtained using large pre-trained 2D foundation models into 3D features, enabling impressive 3D editing and understanding capabilities with only 2D supervision. While powerful, such features contain significant view-dependent components, especially in scenes with complex materials and reflections. When distilled into a single 3D feature field, these inconsistencies are averaged, degrading feature quality and harming downstream tasks like segmentation. We hypothesize that explicitly modeling the physical causes of view-dependence is key to "cleaning" these features during distillation. To this end, we propose to decompose the 3D feature field into view-independent and view-dependent components, guided by a physically-based reflection model. Our core contribution is demonstrating that this structural disentanglement improves the quality and view-invariance of the distilled semantic features. This leads to improved 3D segmentation, particularly in challenging reflective regions, and enables higher-fidelity physically-grounded editing applications. Our project page is available at https://structurallydisentangled.github.io/.

Submission Number: 88