Offline RL Without Off-Policy Evaluation

David Brandfonbrener, William F Whitney, Rajesh Ranganath, Joan Bruna

Keywords: Offline reinforcement learning, reinforcement learning

Abstract: Most prior approaches to offline reinforcement learning (RL) have taken an iterative actor-critic approach involving off-policy evaluation. In this paper we show that simply doing one step of constrained/regularized policy improvement using an on-policy Q estimate of the behavior policy performs surprisingly well. This one-step algorithm beats the previously reported results of iterative algorithms on a large portion of the D4RL benchmark. The one-step baseline achieves this strong performance while being notably simpler and more robust to hyperparameters than previously proposed iterative algorithms. We argue that the relatively poor performance of iterative approaches is a result of the high variance inherent in doing off-policy evaluation and magnified by the repeated optimization of policies against those estimates. In addition, we hypothesize that the strong performance of the one-step algorithm is due to a combination of favorable structure in the environment and behavior policy.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

TL;DR: Performing one step of policy iteration provides a strong baseline for offline RL.

Code: https://github.com/davidbrandfonbrener/onestep-rl

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2106.08909/code)