ClusteringSDF: Self-Organized Neural Implicit Surfaces for 3D Decomposition

Tianhao Wu, Chuanxia Zheng, Qianyi Wu, Tat-Jen Cham

Abstract: 3D decomposition/segmentation remains a challenge as large-scale 3D annotated data is not readily available. Existing approaches typically leverage 2D machine-generated segments, integrating them to achieve 3D consistency. In this paper, we propose ClusteringSDF, a novel approach achieving both segmentation and reconstruction in 3D via the neural implicit surface representation, specifically the Signed Distance Function (SDF), where the segmentation rendering is directly integrated with the volume rendering of neural implicit surfaces. Although based on ObjectSDF++, ClusteringSDF no longer requires ground-truth segments for supervision while maintaining the capability of reconstructing individual object surfaces, relying purely on the noisy and inconsistent labels from pre-trained models. As the core of ClusteringSDF, we introduce a highly efficient clustering mechanism for lifting 2D labels to 3D. Experimental results on the challenging scenes from ScanNet and Replica datasets show that ClusteringSDF  can achieve competitive performance compared to the state-of-the-art with significantly reduced training time.