Correcting Momentum in Temporal Difference Learning

Emmanuel Bengio, Joelle Pineau, Doina Precup

Keywords: Momentum, Reinforcement Learning, Temporal Difference, Deep Reinforcement Learning

Abstract: A common optimization tool used in deep reinforcement learning is momentum, which consists in accumulating and discounting past gradients, reapplying them at each iteration. We argue that, unlike in supervised learning, momentum in Temporal Difference (TD) learning accumulates gradients that become doubly stale: not only does the gradient of the loss change due to parameter updates, the loss itself changes due to bootstrapping. We first show that this phenomenon exists, and then propose a first-order correction term to momentum. We show that this correction term improves sample efficiency in policy evaluation by correcting target value drift. An important insight of this work is that deep RL methods are not always best served by directly importing techniques from the supervised setting.

One-sentence Summary: We add extra terms to momentum to correct for its staleness, this helps TD learning.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2106.03955/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=7oquXh896t