Reusing Historical Trajectories in Natural Policy Gradient via Importance Sampling: Convergence and Convergence Rate

Yifan Lin, Yuhao Wang, Enlu Zhou

Keywords: Importance sampling, reinforcement learning, policy gradient

Abstract:

We study trajectory reuse in natural policy gradient methods.
Classical policy gradient algorithms require large amounts of fresh data, which limits their sample efficiency.
We propose RNPG, a reuse-based natural policy gradient algorithm that incorporates past trajectories through importance weighting of both the gradient and the Fisher information matrix estimators.
We establish asymptotic convergence and a weak convergence rate for RNPG, showing that reuse improves efficiency without altering the limiting behavior.
Experiments on the Cartpole benchmark demonstrate that RNPG achieves faster convergence and smoother performance than VPG and VNPG, with additional gains from larger reuse sizes.
Our results highlight the theoretical and empirical benefits of reusing trajectories in policy optimization.

Submission Number: 12