LAGOON: Language-Guided Motion Control

Shusheng Xu, Huaijie Wang, Yutao Ouyang, Jiaxuan Gao, Zhiyu Mei, Chao Yu, Yi Wu

Keywords: language command, quadrupedal robot, control policy, reinforcement learning

TL;DR: We developed a system called LAGOON. Given a high-level language command, LAGOON can autonomously train a control policy according to the provided command.

Abstract: We aim to control a robot to physically behave in the real world following any high-level language command like “cartwheel” or “kick”. Although human motion datasets exist, this task remains particularly challenging since generative models can produce physically unrealistic motions, which will be more severe for robots due to different body structures and physical properties. Deploying such a motion to a physical robot can cause even greater difficulties due to the sim2real gap. We develop LAnguage-Guided mOtion cONtrol (LAGOON), a multi-phase reinforcement learning (RL) method to generate physically realistic robot motions under language commands. LAGOON first leverages a pre-trained model to generate a human motion from a language command. Then an RL phase trains a control policy in simulation to mimic the generated human motion. Finally, with domain randomization, our learned policy can be deployed to a quadrupedal robot, leading to a quadrupedal robot that can take diverse behaviors in the real world under natural language commands

Submission Number: 5