Behavior From the Void: Unsupervised Active Pre-Training

Hao Liu, Pieter Abbeel

Keywords: unsupervised exploration, unsupervised pretraining, reinforcement learning

TL;DR: We present an unsupervised pre-training RL method that is based on particle-based state entropy maximization and it outperforms canonical RL methods.

Abstract: We introduce a new unsupervised pre-training method for reinforcement learning called APT, which stands for Active Pre-Training. APT learns behaviors and representations by actively searching for novel states in reward-free environments. The key novel idea is to explore the environment by maximizing a non-parametric entropy computed in an abstract representation space, which avoids challenging density modeling and consequently allows our approach to scale much better in environments that have high-dimensional observations (e.g., image observations). We empirically evaluate APT by exposing task-specific reward after a long unsupervised pre-training phase. In Atari games, APT achieves human-level performance on 12 games and obtains highly competitive performance compared to canonical fully supervised RL algorithms. On DMControl suite, APT beats all baselines in terms of asymptotic performance and data efficiency and dramatically improves performance on tasks that are extremely difficult to train from scratch.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/rll-research/url_benchmark

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2103.04551/code)