Constrained Skill Discovery: Quadruped Locomotion with Unsupervised Reinforcement Learning

Vassil Atanassov, Wanming Yu, Alexander Luis Mitchell, Mark Nicholas Finean, Ioannis Havoutis

Keywords: legged locomotion, unsupervised skill discovery, empowerment, unsupervised reinforcement learning

TL;DR: We propose a norm-matching objective and Euclidean distance constraint for unsupervised skill discovery which allows robots to learn diverse locomotive behaviors without task-specific rewards.

Abstract:

Representation learning and unsupervised skill discovery can allow robots to acquire diverse and reusable behaviors without the need for task-specific rewards. In this work, we learn a latent representation by maximizing the mutual information between skills and states subject to a distance constraint, using unsupervised reinforcement learning. Our method improves upon prior constrained skill discovery methods by replacing the latent transition maximization with a norm-matching objective. This not only results in a much a richer state space coverage, but allows the robot to learn more stable and easily controllable locomotive behaviors. In robotics this is particularly important, because state transition-maximizing behaviors can result in highly dangerous motions. We successfully deployed the learned policy on a real ANYmal quadruped robot and demonstrated that the robot can accurately reach arbitrary points of the Cartesian state space in a zero-shot manner, using only an intrinsic skill discovery and standard regularization rewards.

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10970