Reinforcement Learning with Extreme Minimum Distribution
Keywords: Reinforcement Learning, Distributional Reinforcement Learning, Extreme Value Theorem
Abstract: Distributional Reinforcement Learning (DRL) has recently achieved remarkable performance by leveraging the distributional information of accumulated rewards. Because the data in the tail of the distribution is scarce, it is difficult to fit the tail distribution, resulting in the tail behavior of the return distribution that limits the performance of DRL. One solution to tackle the tail behavior of the return distribution is truncating the tail atoms in DRL, but this approach does not have theoretical support in previous work. We adopt the perspective of Extreme Value Theorem, which allows us to reconsider the rationality of truncating tail atoms. We further derive a conclusion: Truncating tail atoms eventually transforms the distribution generated by DRL into the extreme minimum distribution and remains stable. Building on this conclusion, we redesign the critic networks in actor-critic method. Specifically, we use two critics that output the location and scale parameters of the extreme minimum distribution to directly model the distribution of truncated DRL. Our method uses fewer networks and fewer computational resources than other truncated DRL methods and achieves more competitive experimental results than other RL methods.
Supplementary Material: zip
Primary Area: reinforcement learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 2462
Loading