Distilled Feature Fields Enable Few-Shot Language-Guided ManipulationDownload PDF

Published: 30 Aug 2023, Last Modified: 17 Oct 2023CoRL 2023 OralReaders: Everyone
Keywords: Neural Fields, Foundation Models, Scene Understanding, Robot Manipulation
TL;DR: Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation
Abstract: Self-supervised and language-supervised image models contain rich knowledge of the world that is important for generalization. Many robotic tasks, however, require a detailed understanding of 3D geometry, which is often lacking in 2D image features. This work bridges this 2D-to-3D gap for robotic manipulation by leveraging distilled feature fields to combine accurate 3D geometry with rich semantics from 2D foundation models. We present a few-shot learning method for 6-DOF grasping and placing that harnesses these strong spatial and semantic priors to achieve in-the-wild generalization to unseen objects. Using features distilled from a vision-language model, CLIP, we present a way to designate novel objects for manipulation via free-text natural language, and demonstrate its ability to generalize to unseen expressions and novel categories of objects. Project website: https://f3rm.csail.mit.edu
Student First Author: yes
Supplementary Material: zip
Instructions: I have read the instructions for authors (https://corl2023.org/instructions-for-authors/)
Video: https://www.youtube.com/watch?v=PA9rWWVWsc4
Website: https://f3rm.csail.mit.edu
Code: https://github.com/f3rm/f3rm
Publication Agreement: pdf
10 Replies