Guaranteed Trust Region Optimization via Two-Phase KL Penalization

K.R. Zentner, Ujjwal Puri, Zhehui Huang, Gaurav S. Sukhatme

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Reinforcement Learning, Trust Region Methods, On-Policy RL

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Adding a small fixup pass to KL penalized on-policy RL can guarantee constrained updates and requires very few gradient steps.

Abstract:

On-policy reinforcement learning (RL) has become a popular framework for solving sequential decision problems due to its computational efficiency and theoretical simplicity.
Some on-policy methods guarantee every policy update is constrained to a trust region relative to the prior policy to ensure training stability.
These methods often require computationally intensive non-linear optimization or require a particular form of action distribution.
In this work, we show that applying KL penalization alone is nearly sufficient to enforce such trust regions.
Then, we show that introducing a "fixup" phase is sufficient to guarantee a trust region is enforced on every policy update while adding fewer than 5\% additional gradient steps in practice.
The resulting algorithm, which we call FixPO, is able to train a variety of policy architectures and action spaces, is easy to implement, and produces results competitive with other trust region methods.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7070