Deep Reinforcement Learning (RL) has emerged as a powerful paradigm to solve a range of complex yet specific control tasks. Training generalist agents that can quickly adapt to new tasks remains an outstanding challenge. Recent advances in unsupervised RL have shown that pre-training RL agents with self-supervised intrinsic rewards can result in efficient adaptation. However, these algorithms have been hard to compare and develop due to the lack of a unified benchmark. To this end, we introduce the Unsupervised Reinforcement Learning Benchmark (URLB). URLB consists of two phases: reward-free pre-training and downstream task adaptation with extrinsic rewards. Building on the DeepMind Control Suite, we provide twelve continuous control tasks from three domains for evaluation and open-source code for eight leading unsupervised RL methods. We find that the implemented baselines make progress but are not able to solve URLB and propose directions for future research.


URLB: Unsupervised Reinforcement Learning Benchmark

In exceptional cases, e.g. for ethical reasons, datasets can be released under an established open credentialized access procedure, (e.g. PhysioNet credentialing for use of human subject data). If this is a case, briefly describe here how to access the data. This field is publicly visible, hence not meant for sending a private access link to reviewers, you can do that after the initial submission by sending an official note to the reviewers. Please see the Call For Papers for more details. If unsure, leave this field open.

"Too Long; Didn't Read": a short sentence describing your paper

Public URL link to dataset or benchmark (if applicable). Instructions on how to access may also be provided here. If the dataset or benchmark needs to be private from the public during the review process, please provide the URL link and any access information instead using the 'Add Dataset Benchmark URL' button that will become visible on the paper webpage, after a paper is submitted. Information added through this button will be only visible to reviewers / ACs /PCs. This must also be completed by the paper submission deadline.

Abstract of paper. Add TeX formulas using the following formats: $In-line Formula$ or $$Block Formula$$

As an author, you confirm that you bear all responsibility in case of any violation of rights during the collection of the data or other work, and will take appropriate action when needed, e.g. to remove data with such issues.

Search author profile by first, middle and last name or email address. If the profile is not found, you can add the author by completing first, middle, and last names as well as author email address.

I have read the detailed guidelines for submission to the NeurIPS 2021 Datasets and Benchmarks Track in the call for papers, as well as the email providing additional guidelines for camera-ready copies, and agree with and adhere to these fully.

In select cases, a dataset can be released publicly at a later date, but no longer than 1 year after the submission deadline. These papers will be marked in the online proceedings. If this is the case, provide an explanation below and the date of release.

For datasets, provide the URL of the dataset or a landing page with clear information on how to access the dataset.

If your submission includes a dataset or code, state the license under which data or work is released.

Supplementary material. All supplementary material must be self-contained and submitted as a PDF or multiple files zipped into a single file. Access to datasets and benchmarks may be provided via the URL field below.

Title of paper. Add TeX formulas using the following formats: $In-line Formula$ or $$Block Formula$$

URLB: Unsupervised Reinforcement Learning Benchmark

Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, Pieter Abbeel