SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields

Anthony Simeonov, Yilun Du, Yen-Chen Lin, Alberto Rodriguez Garcia, Leslie Pack Kaelbling, Tomás Lozano-Pérez, Pulkit Agrawal

Keywords: Neural Fields, Relations, Manipulation, Object Rearrangement

TL;DR: We present a method to execute relational rearrangement tasks on pairs of unseen objects using a few demonstrations.

Abstract: We present a framework for specifying tasks involving spatial relations between objects using only 5-10 demonstrations and then executing such tasks given point cloud observations of a novel pair of objects in arbitrary initial poses. Our approach structures these rearrangement tasks by assigning a consistent local coordinate frame to the task-relevant object parts, localizing the corresponding coordinate frame on unseen object instances, and executing an action that brings these frames into alignment. We propose an optimization method that uses multiple Neural Descriptor Fields (NDFs) and a single annotated 3D keypoint to assign a set of consistent coordinate frames to the task-relevant object parts. We also propose an energy-based learning scheme to model the joint configuration of the objects that satisfies a desired relational task. We validate our pipeline on three multi-object rearrangement tasks in simulation and on a real robot. Results show that our method can infer relative transformations that satisfy the desired relation between novel objects in unseen initial poses using just a few demonstrations.

Student First Author: yes

Website: https://anthonysimeonov.github.io/r-ndf/

Code: https://github.com/anthonysimeonov/relational_ndf

Community Implementations: [ 4 code implementations](https://www.catalyzex.com/paper/arxiv:2211.09786/code)