Iterative Amortized Policy Optimization

Joseph Marino, Alexandre Piché, Alessandro Davide Ialongo, Yisong Yue

Keywords: Reinforcement Learning, Policy Optimization, Amortization, Variational Inference

Abstract: Policy networks are a central feature of deep reinforcement learning (RL) algorithms for continuous control, enabling the estimation and sampling of high-value actions. From the variational inference perspective on RL, policy networks, when employed with entropy or KL regularization, are a form of amortized optimization, optimizing network parameters rather than the policy distributions directly. However, this direct amortized mapping can empirically yield suboptimal policy estimates and limited exploration. Given this perspective, we consider the more flexible class of iterative amortized optimizers. We demonstrate that the resulting technique, iterative amortized policy optimization, yields performance improvements over direct amortization methods on benchmark continuous control tasks.

One-sentence Summary: Policy networks in RL are direct amortized optimizers, and we demonstrate the benefits of using more flexible iterative amortized optimizers.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:2010.10670/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=0DIDoyIs6V