Generalization through hand-eye coordination: An action space for learning spatially-invariant visuomotor control

Rui Wang

Abstract:

Abstract— Imitation Learning (IL) is an effective framework
to learn visuomotor skills from offline demonstration data.
However, IL methods often fail to generalize to new scene
configurations not covered by training data. On the other hand,
humans can manipulate objects in varying conditions. Key to
such capability is hand-eye coordination, a cognitive ability
that enables humans to adaptively direct their movements at
task-relevant objects and be invariant to the objects’ absolute
spatial location. In this work, we present a learnable action
space, Hand-eye Action Networks (HAN), that can approximate human’s hand-eye coordination behaviors by learning
from human teleoperated demonstrations. Through a set of
challenging multi-stage manipulation tasks, we show that a
visuomotor policy equipped with HAN is able to inherit the
key spatial invariance property of hand-eye coordination and
achieve zero-shot generalization to new scene configurations.
Additional materials available at https://sites.google.
com/stanford.edu/han