Beyond Average Reward in Markov Decision ProcessesDownload PDF

Published: 20 Jul 2023, Last Modified: 31 Aug 2023EWRL16Readers: Everyone
Keywords: Markov Decision Process, Dynamic Programming, statistical functionnals, Distributionnal Reinforcement Learning, Policy Evaluation, Planning
Abstract: What are the functionals of the reward that can be computed and optimized exactly in Markov Decision Processes? In the finite-horizon, undiscounted setting, Dynamic Programming (DP) can only handle these operations efficiently for certain classes of statistics. We summarize the characterization of these classes for policy evaluation, and give a new answer for the planning problem. Interestingly, we prove that only generalized means can be optimized exactly. It is possible, however, to evaluate other functionals approximately using Distributional Reinforcement Learning. We prove error bounds on the resulting estimators and discuss the potential and limitations of this approach. These results contribute to advancing the theory of Markov Decision Processes by examining overall characteristics of the return, and particularly risk-conscious strategies.
1 Reply