Learning Generalizable Manipulation Policies with Object-Centric 3D RepresentationsDownload PDF

Published: 30 Aug 2023, Last Modified: 22 Oct 2023CoRL 2023 PosterReaders: Everyone
Keywords: robot manipulation, imitation learning, object-centric representations
TL;DR: We introduce an imitation learning method for training policies that generalize beyond their demonstration settings.
Abstract: We introduce GROOT, an imitation learning method for learning robust policies with object-centric and 3D priors. GROOT builds policies that generalize beyond their initial training conditions for vision-based manipulation. It constructs object-centric 3D representations that are robust toward background changes and camera views and reason over these representations using a transformer-based policy. Furthermore, we introduce a segmentation correspondence model that allows policies to generalize to new objects at test time. Through comprehensive experiments, we validate the robustness of GROOT policies against perceptual variations in simulated and real-world environments. GROOT’s performance excels in generalization over background changes, camera viewpoint shifts, and the presence of new object instances, whereas both state-of-the-art end-to-end learning methods and object proposal-based approaches fall short. We also extensively evaluate GROOT policies on real robots, where we demonstrate the efficacy under very wild changes in setup. More videos and model details can be found in the appendix and the project website https://ut-austin-rpl.github.io/GROOT.
Student First Author: yes
Supplementary Material: zip
Instructions: I have read the instructions for authors (https://corl2023.org/instructions-for-authors/)
Website: https://ut-austin-rpl.github.io/GROOT/
Code: https://github.com/UT-Austin-RPL/GROOT
Publication Agreement: pdf
Poster Spotlight Video: mp4
12 Replies