Iterative Amortized Policy Optimization

Joseph Marino, Alexandre Piché, Alessandro Davide Ialongo, Yisong Yue

Keywords: Amortization, Variational Inference, Reinforcement Learning

TL;DR: We connect KL-regularized policy networks with amortization and propose an iterative amortized policy optimization scheme with novel benefits and performance improvements.

Abstract: Policy networks are a central feature of deep reinforcement learning (RL) algorithms for continuous control, enabling the estimation and sampling of high-value actions. From the variational inference perspective on RL, policy networks, when used with entropy or KL regularization, are a form of amortized optimization, optimizing network parameters rather than the policy distributions directly. However, direct amortized mappings can yield suboptimal policy estimates and restricted distributions, limiting performance and exploration. Given this perspective, we consider the more flexible class of iterative amortized optimizers. We demonstrate that the resulting technique, iterative amortized policy optimization, yields performance improvements over direct amortization on benchmark continuous control tasks.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/joelouismarino/variational_rl

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:2010.10670/code)