Reward Shifting for Optimistic Exploration and Conservative Exploitation

Hao Sun, Lei Han, Jian Guo, Bolei Zhou

Keywords: Reward Shift, Reinforcement Learning, Batch RL, Offline RL, Online RL, Curiosity-Driven Method

Abstract: In this work, we study the simple yet universally applicable case of reward shaping, the linear transformation, in value-based Deep Reinforcement Learning. We show that reward shifting, as the simplest linear reward transformation, is equivalent to changing initialization of the $Q$-function in function approximation. Based on such an equivalence, we bring the key insight that a positive reward shifting leads to conservative exploitation, while a negative reward shifting leads to curiosity-driven exploration. In this case, a conservative exploitation improves offline RL value estimation, and the optimistic value estimation benefits the exploration of online RL. We verify our insight on a range of tasks: (1) In offline RL, the conservative exploitation leads to improved learning performance based on off-the-shelf algorithms; (2) In online continuous control, multiple value functions with different shifting constants can be used to trade-off between exploration and exploitation thus improving learning efficiency; (3) In online RL with discrete action space, a negative reward shifting brings an improvement over the previous curiosity-based exploration method.

One-sentence Summary: Linear reward transformations are equivalent to different initializations the $Q$-function for value-based RL and can be used for conservative exploitation as well as curiosity-driven exploration.