NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning

Diyuan Shi, Tang Yiqi, Jinxin Liu, Zifeng Zhuang, Donglin Wang

Keywords: Reinforcement Learning, Policy Optimization

Abstract: Deep Reinforcement Learning (DRL) has experienced significant advancements in recent years and has been widely used in many fields. In DRL-based robotic policy learning, however, current *de facto* policy parameterization is still multivariate Gaussian (with diagonal covariance matrix), which lacks the ability to model multi-modal distribution. In this work, we explore the adoption of a modern network architecture, i.e. Normalizing Flow (NF) as the policy parameterization for its ability of multi-modal modeling, closed form of log probability and low computation and memory overhead. However, naively training NF in online Reinforcement Learning (RL) usually leads to training instability. We provide a detailed analysis for this phenomenon and successfully address it via simple but effective technique. With extensive experiments in multiple simulation environments, we show our method, NFPO could obtain robust and strong performance in widely used robotic learning tasks and successfully transfer into real-world robots.

Primary Area: reinforcement learning

Submission Number: 8345