Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learning

Hyunwoo Ryu, Hong-in Lee, Jeong-Hoon Lee, Jongeun Choi

Keywords: Robotics, Manipulation, Robotic manipulation, Equivariance, SE(3), SO(3), Energy-based model, Lie group, Representation theory, Equivariant robotics, Roto-translation equivariance, End-to-end, Point clouds, Graph neural networks, Imitation learning, Learning from demonstration, Sample efficient, Few shot, Unseen object, Category-level manipulation, MCMC, Langevin dynamics

TL;DR: We present SE(3)-equivariant energy based models that can learn robotic manipulation tasks end-to-end from only few demonstrations without any prior knowledge.

Abstract: End-to-end learning for visual robotic manipulation is known to suffer from sample inefficiency, requiring large numbers of demonstrations. The spatial roto-translation equivariance, or the SE(3)-equivariance can be exploited to improve the sample efficiency for learning robotic manipulation. In this paper, we present SE(3)-equivariant models for visual robotic manipulation from point clouds that can be trained fully end-to-end. By utilizing the representation theory of the Lie group, we construct novel SE(3)-equivariant energy-based models that allow highly sample efficient end-to-end learning. We show that our models can learn from scratch without prior knowledge and yet are highly sample efficient (5~10 demonstrations are enough). Furthermore, we show that our models can generalize to tasks with (i) previously unseen target object poses, (ii) previously unseen target object instances of the category, and (iii) previously unseen visual distractors. We experiment with 6-DoF robotic manipulation tasks to validate our models' sample efficiency and generalizability. Codes are available at: https://github.com/tomato1mule/edf

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Applications (eg, speech processing, computer vision, NLP)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2206.08321/code)