Leveraging Demonstrations with Latent Space Priors

Published: 13 Mar 2023, Last Modified: 13 Mar 2023Accepted by TMLREveryoneRevisionsBibTeX
Abstract: Demonstrations provide insight into relevant state or action space regions, bearing great potential to boost the efficiency and practicality of reinforcement learning agents. In this work, we propose to leverage demonstration datasets by combining skill learning and sequence modeling. Starting with a learned joint latent space, we separately train a generative model of demonstration sequences and an accompanying low-level policy. The sequence model forms a latent space prior over plausible demonstration behaviors to accelerate learning of high-level policies. We show how to acquire such priors from state-only motion capture demonstrations and explore several methods for integrating them into policy learning on transfer tasks. Our experimental results confirm that latent space priors provide significant gains in learning speed and final performance. We benchmark our approach on a set of challenging sparse-reward environments with a complex, simulated humanoid, and on offline RL benchmarks for navigation and object manipulation.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: Camera ready
Code: https://facebookresearch.github.io/latent-space-priors
Assigned Action Editor: ~Caglar_Gulcehre1
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Number: 523