Extending Differential Temporal Difference Methods to Episodic Problems

ICLR 2026 Conference Submission16813 Authors

Keywords: reinforcement learning, temporal difference, streamingnormalization, average reward

TL;DR: We generalize differential TD—an infinite-horizon solution method that relies on reward centering—to maintain the ordering of policies in episodic problems. We empirically show that reward centering can improve sample efficiency in episodic problems.

Abstract: Differential temporal difference (TD) methods are value-based reinforcement learning algorithms that have been proposed for infinite-horizon problems. They rely on reward centering, where each reward is centered by the average reward. This keeps the return bounded and removes a value function’s state-independent offset. However, reward centering can alter the optimal policy in episodic problems, limiting its applicability. Motivated by recent works that emphasize the role of normalization in streaming deep reinforcement learning, we study reward centering in episodic problems and propose a generalization of differential TD. We prove that this generalization maintains the ordering of policies in the presence of termination, and thus extends differential TD to episodic problems. We show equivalence with a form of linear TD, thereby inheriting theoretical guarantees that have been shown for those algorithms. We then extend several streaming reinforcement learning algorithms to their differential counterparts. Across a range of base algorithms and environments, we empirically validate that reward centering can improve sample efficiency in episodic problems.

Primary Area: reinforcement learning

Submission Number: 16813