On Temporal Credit Assignment and Data-Efficient Reinforcement Learning

Dilip Arumugam, Thomas L. Griffiths

Keywords: Temporal Credit Assignment, Information Theory, Data-Efficient RL

TL;DR: Through the introduction of a novel performance criterion for RL algorithms, we offer one notion of what it means to achieve statistically-efficient credit assignment.

Abstract: The challenge of temporal credit assignment in reinforcement learning (RL) can be articulated as a simple question about the behavior of a sequential decision-making agent: how does the execution of particular actions from specific states impact observed future outcomes? Typically, one asks this question for each state-action pair sampled along a full trajectory within the environment and the future outcome of interest is the cumulative return obtained by an agent. Temporal credit assignment stands as the defining challenge of the RL paradigm, distinguishing it from supervised learning and bandit learning settings, where the data-efficiency challenges of generalization and exploration also arise. Nevertheless, a precise and formal characterization of the credit assignment problem remains elusive. In this work, we make an initial effort to formally define the credit assignment problem through the introduction of a performance measure for RL algorithms, quantifying the overall accuracy of credit attribution (or lack thereof) between the policies generated by an RL algorithm and the optimal policy. To define this novel performance criterion, we draw upon foundational information-theoretic and game-theoretic tools for the partial decomposition of information and the allocation of group compensation among individual team members.

Submission Number: 24