Zero-Incentive Dynamics: a look at reward sparsity through the lens of unrewarded subgoals

Yannick Molinghen, Tom Lenaerts

Keywords: Reinforcement Learning, Reward sparsity, Multi-Agent Systems

TL;DR: We show that reward sparsity alone does not explain task difficulty in RL, introduce Zero-Incentive Dynamics to account for unrewarded subtasks and show the important of subtask-to-reward proximity.

Abstract: This work re-examines the commonly held assumption that the frequency of rewards is a sufficient measure of task difficulty in reinforcement learning. We identify and formalize a structural challenge that undermines the effectiveness of current policy learning methods: when essential subgoals do not directly yield rewards. We characterize such settings as exhibiting zero-incentive dynamics, where transitions critical to success remain unrewarded. We show that state-of-the-art deep subgoal-based algorithms fail to leverage these dynamics and that learning performance is highly sensitive to the temporal proximity between subgoal completion and eventual reward. These findings reveal a fundamental limitation in current approaches and point to the need for mechanisms that can infer latent task structure without relying on immediate incentives.

Submission Number: 32