Language Conditioned Equivariant Grasp

Haojie Huang, Mingxi Jia, Zhewen Zhang, Ondrej Biza, Linfeng Zhao, Robin Walters, Robert Platt

Primary Area: applications to robotics, autonomy, planning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Robot Learning, Geometric Deep Learning, Robotic Manipulation, Equivariant Deep Learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: The ability to control robots with simple natural language instructions enables non-experts to employ robots as general tools and has long been a goal in robot learning. In this paper, we examine the problem of training a robotic grasping policy conditioned on language instructions. This is inherently challenging since efficient manipulation policy learning often exploits symmetry and geometry in the task, but it is unclear how to incorporate language into such a framework. In this work, we present $\text{L}$anguage-conditioned $\text{E}$quivariant $\text{G}$rasp ($\text{LEG}$), which leverages the $\mathrm{SE}(2)$ symmetries of language-conditioned robotic grasping by mapping the language instruction to an $\mathrm{SO}(2)$-steerable kernel. We demonstrate the sample efficiency and performance of this method on the Language-Grasp Benchmark which includes 10 different language-conditioned grasping tasks and evaluate it on a real robot.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5778