Non-Local Data Attribution for On-policy Reinforcement Learning

Shixuan Liu, Yuzheng Hu, Han Zhao, Jiaqi W. Ma

Keywords: Data-centric AI, Data Attribution, Influence Estimation, On-policy Reinforcement Learning

TL;DR: We introduce non-local data attribution for reinforcement learning, and show that existing attribution methods can accurately approximate replay-LOO effects and enable Lookahead Iterative Filtering (LIF) to improve on-policy RL training.

Abstract: Data attribution has become an important tool for understanding and improving model training, but its study in reinforcement learning (RL) remains limited. Prior work has shown that local data attribution computed within a single rollout provides useful signals for data selection and hence helps accelerate training. In this work, we move beyond local attribution and introduce non-local data attribution for on-policy RL, where attribution targets are defined using future rollouts generated by a better-performing policy. We formalize this setting via a replay-based leave-one-out objective (replay-LOO) that isolates optimization effects under fixed rollout buffers. Using the well-developed training data attribution methods in supervised learning, we are able to account for the training dynamics when estimating data influence. We show that non-local attribution achieves strong correlation with ground-truth LOO retraining effects in RL. Based on this property, we further demonstrate how non-local attribution can be used for effective data selection by reusing rollout buffers, leading to improved sample efficiency without additional environment interaction. Overall, our results highlight non-local attribution as a promising tool for data-centric reinforcement learning.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 134