SeGS: Semantic-aware 3D Gaussian Splatting for Multi-turn Language-guided Robotic Grasping in Changeable Environment
Abstract: In real-world applications, robots often need to perform multi-turn grasping, which requires them to accurately perceive and adapt to changeable environments. Current approaches extract complex semantics from 2D foundational models to implicitly represent 3D scenes. However, these methods necessitate a complete scene re-learning, leading to significant delays and inefficiencies in handling updated environments. To address this limitation, this paper introduces Semantic-aware 3D Gaussians Splatting (SeGS), a novel method that explicitly represents scenes with rich semantic information, enabling rapid scene updating in evolving environments. SeGS incorporates 3D Gaussian Splatting(3DGS) and integrates semantic features into each 3D Gaussian to capture contextual semantic details. By using explicit 3DGS and employing rend-and-compare strategy, SeGS allows for fast adaptation to scene changes, equipping robots to execute multi-turn grasping in changeable environments. Extensive experiments on continuous tasks demonstrate SeGS’s ability to quickly reconstruct altered scenarios, facilitating swift task execution.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Wanli_Ouyang1
Submission Number: 3794
Loading