The Difficulty of Passive Learning in Deep Reinforcement LearningDownload PDF

21 May 2021, 20:42 (edited 26 Oct 2021)NeurIPS 2021 PosterReaders: Everyone
  • Keywords: deep reinforcement learning, deep RL, offline reinforcement learning, offline RL, DQN, reinforcement learning, off-policy, policy improvement, policy optimization, distributional shift, empirical analysis, empirical method
  • TL;DR: Empirical investigation of the importance of learning from one's own mistakes in deep reinforcement learning
  • Abstract: Learning to act from observational data without active environmental interaction is a well-known challenge in Reinforcement Learning (RL). Recent approaches involve constraints on the learned policy or conservative updates, preventing strong deviations from the state-action distribution of the dataset. Although these methods are evaluated using non-linear function approximation, theoretical justifications are mostly limited to the tabular or linear cases. Given the impressive results of deep reinforcement learning, we argue for a need to more clearly understand the challenges in this setting. In the vein of Held & Hein's classic 1963 experiment, we propose the "tandem learning" experimental paradigm which facilitates our empirical analysis of the difficulties in offline reinforcement learning. We identify function approximation in conjunction with fixed data distributions as the strongest factors, thereby extending but also challenging hypotheses stated in past work. Our results provide relevant insights for offline deep reinforcement learning, while also shedding new light on phenomena observed in the online case of learning control.
  • Supplementary Material: pdf
  • Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
  • Code: https://github.com/deepmind/deepmind-research/tree/master/tandem_dqn
10 Replies

Loading