Reward Shaping Control Variates for Off-Policy Evaluation Under Sparse Rewards

ICLR 2026 Conference Submission13687 Authors

Keywords: reward shaping, potentials, off-policy evaluation, bias, variance

TL;DR: We show how potential-based reward shaping can be leveraged as a control variate in OPE and provide theoretical and empirical performance guarantees

Abstract: Off-policy evaluation (OPE) is essential for deploying reinforcement learning in safety-critical settings, yet existing estimators such as importance sampling and doubly robust (DR) often exhibit prohibitively high variance when rewards are sparse. In this work, we introduce Reward-Shaping Control Variates, a new family of unbiased estimators that leverage potential-based reward shaping to construct additional zero-mean control variates. We prove that shaped estimators always yields valid variance reduction, and that combining shaping-based and Q-based control variates strictly expands the variance-reduction subspace beyond DR and its minimax variant MRDR. Empirically, we provide a systematic regime map across synthetic chains, a cancer simulator, and an ICU-sepsis benchmark showing that shaping-based OPE consistently outperforms DR in sparse-reward settings, while a hybrid estimator achieves state-of-the-art performance across sparse, noisy, and misspecified environments. Our results highlight reward shaping as a powerful and interpretable tool for robust OPE, offering both theoretical guarantees and practical improvements in domains where standard estimators fail.

Primary Area: reinforcement learning

Submission Number: 13687