Reward Centering
Primary Area: reinforcement learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: reinforcement learning; discounting; average reward
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: We show that discounted methods for solving continuing reinforcement learning problems can be significantly improved if they center their rewards by subtracting out the rewards' (changing) empirical average. The improvement is substantial at commonly-used discount factors and increases further as the discount factor approaches 1. In addition, we show that if a problem's rewards are shifted by a constant, then non-centering methods perform much worse, whereas centering methods are (unsurprisingly) unaffected. In this sense, reward centering significantly increases the generality of discounted reinforcement learning methods. Insight into the benefits of reward centering can be gained from the decomposition of the discounted value function proposed by Blackwell in 1962.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6417
Loading