Improving Reward-Based Hindsight Credit Assignment

Aditya A. Ramesh, Jiamin He, Jürgen Schmidhuber, Martha White

Keywords: temporal credit assignment, hindsight credit assignment, model-based reinforcement learning, sample-efficiency

Abstract: Accurately attributing credit or blame for outcomes to past actions is crucial for sample-efficient reinforcement learning. While temporal-difference learning with $\lambda$-returns is the most commonly used approach, it attributes credit based on the temporal proximity of actions and outcomes--a heuristic that may be overly simplistic in complex environments. Hindsight-based approaches offer an alternative by using a model that leverages future information to more explicitly credit previous actions that were critical to achieving specific outcomes. Recent work has shown that predicting past actions using future rewards can be effective for hindsight credit assignment in Markovian environments. However, we show that the associated credit-assignment algorithm suboptimally handles immediate rewards, potentially resulting in high variance even with perfect hindsight models. We introduce a simple correction that resolves this issue.

Confirmation: I understand that authors of each paper submitted to EWRL may be asked to review 2-3 other submissions to EWRL.

Serve As Reviewer: ~Aditya_A._Ramesh1

Track: Regular Track: unpublished work

Submission Number: 137