Adversarial Object Rearrangement in Constrained Environments with Heterogeneous Graph Neural Networks

Xibai Lou, Houjian Yu, Ross Worobel, Yang Yang, Changhyun Choi

Abstract:

Adversarial object rearrangement in the real
world (e.g., previously unseen or oversized items in kitchens
and stores) could benefit from understanding task scenes,
which inherently entail heterogeneous components such as
current objects, goal objects, and environmental constraints.
The semantic relationships among these components are distinct
from each other and crucial for multi-skilled robots to perform
efficiently in everyday scenarios. We propose a hierarchical
robotic manipulation system that learns the underlying relation-
ships and maximizes the collaborative power of its diverse skills
(e.g., PICK-PLACE, PUSH) for rearranging adversarial objects in
constrained environments. The high-level coordinator employs a
heterogeneous graph neural network (HetGNN), which reasons
about the current objects, goal objects, and environmental
constraints; the low-level 3D Convolutional Neural Network-
based actors execute the action primitives. Our approach is
trained entirely in simulation, and achieved an average success
rate of 87.88% and a planning cost of 12.82 in real-world
experiments, surpassing all baseline methods. Supplementary
material is available at https://sites.google.com/umn.
edu/versatile-rearrangement.