An empirical study of implicit regularization in deep offline RL

Caglar Gulcehre, Srivatsan Srinivasan, Jakub Sygnowski, Georg Ostrovski, Mehrdad Farajtabar, Matthew Hoffman, Razvan Pascanu, Arnaud Doucet

Abstract: Deep neural networks are the most commonly used function approximators in offline reinforcement learning. Prior works have shown that neural nets trained with TD-learning and gradient descent can exhibit implicit regularization that can be characterized by under-parameterization of these networks. Specifically, the rank of the penultimate feature layer, also called effective rank, has been observed to drastically collapse during the training. In turn, this collapse has been argued to reduce the model's ability to further adapt in later stages of learning, leading to the diminished final performance. Such an association between the effective rank and performance makes effective rank compelling for offline RL, primarily for offline policy evaluation. In this work, we conduct a careful empirical study on the relation between effective rank and performance on three offline RL datasets : bsuite, Atari, and DeepMind lab. We observe that a direct association exists only in restricted settings and disappears in the more extensive hyperparameter sweeps. Also, we empirically identify three phases of learning that explain the impact of implicit regularization on the learning dynamics and found that bootstrapping alone is insufficient to explain the collapse of the effective rank. Further, we show that several other factors could confound the relationship between effective rank and performance and conclude that studying this association under simplistic assumptions could be highly misleading.

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

The changes are minimal, focusing on clarifying some of the concerns raised by the reviewers. We list some of them below:

1) The reviewers were confused about how we define complex and simple behaviors. We added a few sentences to the section 4.1 to further clarify this.
2) In the caption of Figure 5 and rephrase the statement about almost like a Gaussian to address the concern raised by reviewer hEii.
3) Added a sentence to Section 2.1 as well about different rank measures that we have run preliminary experiments with.

Assigned Action Editor: ~Nadav_Cohen1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 374