3D-GRES: Generalized 3D Referring Expression Segmentation

Changli Wu, Yihang Liu, Yiwei Ma, Haowei Wang, Gen Luo, Jiayi Ji, Henghui Ding, Xiaoshuai Sun, Rongrong Ji

Abstract: 3D Referring Expression Segmentation (3D-RES) is dedicated to segmenting a specific instance within a 3D space based on a natural language description. However, current approaches are limited to segmenting a single target, restricting the versatility of the task. To overcome this limitation, we introduce Generalized 3D Referring Expression Segmentation (3D-GRES), which extends the capability to segment any number of instances based on natural language instructions. In addressing this broader task, we propose the Multi-Query Decoupled Interaction Network (MDIN), designed to break down multi-object segmentation tasks into simpler, individual segmentations. MDIN comprises two fundamental components: Text-driven Sparse Queries (TSQ) and Multi-object Decoupling Optimization (MDO). TSQ generates sparse point cloud features distributed over key targets as the initialization for queries. Meanwhile, MDO is tasked with assigning each target in multi-object scenarios to different queries while maintaining their semantic consistency. To adapt to this new task, we build a new dataset, namely Multi3DRes. Our comprehensive evaluations on this dataset demonstrate substantial enhancements over existing models, thus charting a new path for intricate multi-object 3D scene comprehension. The benchmark and code are available at https://github.com/sosppxo/MDIN.

Primary Subject Area: [Experience] Multimedia Applications

Relevance To Conference: In the current landscape of multimodal learning, there is growing interest in applying these technologies to more practical, real-world 3D scenarios. This paper introduces a novel task, 3D-GRES, which integrates 3D vision and linguistic techniques, marking a significant advance in this direction. By doing so, it broadens the horizons of multimodal tasks and significantly enhances the impact of multimodal technologies. The methodologies proposed in this paper hold substantial commercial and research value, directly influencing fields such as autonomous driving and robotic navigation. This work not only promotes a deeper understanding of 3D vision but also propels the development of Embodied AI in the upcoming intelligent era.

Submission Number: 586