Splat-MOVER: Multi-Stage, Open-Vocabulary Robotic Manipulation via Editable Gaussian Splatting

Olaolu Shorinwa, Johnathan Tucker, Aliyah Smith, Aiden Swann, Timothy Chen, Roya Firoozi, Monroe David Kennedy, Mac Schwager

Keywords: Gaussian Splatting, Robotic Grasping, Robotic Manipulation, Scene Editing

TL;DR: We present Splat-MOVER, a modular robotics stack for open-vocabulary robotic manipulation, which leverages the editability of Gaussian Splatting scene representations to enable multi-stage manipulation tasks.

Abstract:

We present Splat-MOVER, a modular robotics stack for open-vocabulary
robotic manipulation, which leverages the editability of Gaussian Splatting (GSplat)
scene representations to enable multi-stage manipulation tasks. Splat-MOVER
consists of: (i) ASK-Splat, a GSplat representation that distills semantic and grasp
affordance features into the 3D scene. ASK-Splat enables geometric, semantic,
and affordance understanding of 3D scenes, which is critical for many robotics
tasks; (ii) SEE-Splat, a real-time scene-editing module using 3D semantic masking
and infilling to visualize the motions of objects that result from robot interactions
in the real-world. SEE-Splat creates a “digital twin” of the evolving environment
throughout the manipulation task; and (iii) Grasp-Splat, a grasp generation module
that uses ASK-Splat and SEE-Splat to propose affordance-aligned candidate grasps
for open-world objects. ASK-Splat is trained in real-time from RGB images
in a brief scanning phase prior to operation, while SEE-Splat and Grasp-Splat
run in real-time during operation. We demonstrate the superior performance of
Splat-MOVER in hardware experiments on a Kinova robot compared to two recent
baselines in four single-stage, open-vocabulary manipulation tasks. In addition, we
demonstrate Splat-MOVER in four multi-stage manipulation tasks, using the edited
scene to reflect changes due to prior manipulation stages, which is not possible
with existing baselines. Video demonstrations and the code for the project are
available at https://splatmover.github.io.

Video: https://splatmover.github.io

Website: https://splatmover.github.io

Code: https://splatmover.github.io

Student Paper: yes

Submission Number: 173