EquivAct: SIM(3)-Equivariant Visuomotor Policies beyond Rigid Object Manipulation

Jingyun Yang, Congyue Deng, Jimmy Wu, Rika Antonova, Leonidas Guibas, Jeannette Bohg

Keywords: Generalization in robot learning, domain adaptation, equivariance

TL;DR: We propose EquivAct which utilizes SIM(3)-equivariant network structures that guarantee out-of-distribution generalization of robot manipulation policies across object translations, 3D rotations, and scales by construction.

Abstract: If a robot masters folding a kitchen towel, we would also expect it to master folding a beach towel. However, existing works for policy learning that rely on data set augmentations are still limited in achieving this level of generalization. We propose EquivAct which utilizes SIM(3)-equivariant network structures that guarantee out-of-distribution generalization across object translations, 3D rotations, and scales by construction. Our method first pre-trains a SIM(3)-equivariant visual representation on simulated scene point clouds and then learns a SIM(3)-equivariant visuomotor policy on top of the pre-trained visual representation using a small amount of source task demonstrations. In both simulation and real robot experiments, we show that the learned policy directly transfers to objects that substantially differ in scale, position, and orientation from the source demonstrations. Website: https://equivact.github.io

Submission Number: 7