Zeroth-Order Actor-Critic

Yuheng Lei, Jianyu Chen, Shengbo Eben Li, Sifa Zheng

Keywords: reinforcement learning, zeroth-order optimization, actor-critic

Abstract: Evolution based zeroth-order optimization methods and policy gradient based first-order methods are two promising alternatives to solve reinforcement learning (RL) problems with complementary advantages. The former work with arbitrary policies, drive state-dependent and temporally-extended exploration, possess robustness-seeking property, but suffer from high sample complexity, while the latter are more sample efficient but restricted to differentiable policies and the learned policies are less robust. We propose Zeroth-Order Actor-Critic algorithm (ZOAC) that unifies these two methods into an on-policy actor-critic architecture to preserve the advantages from both. ZOAC conducts rollouts collection with timestep-wise perturbation in parameter space, first-order policy evaluation (PEV) and zeroth-order policy improvement (PIM) alternately in each iteration. The modified rollouts collection strategy and the introduced critic network help to reduce the variance of zeroth-order gradient estimators and improve the sample efficiency and stability of the algorithm. We evaluate our proposed method using two different types of policies, linear policies and neural networks, on a range of challenging continuous control benchmarks, where ZOAC outperforms zeroth-order and first-order baseline algorithms.

One-sentence Summary: We propose Zeroth-Order Actor-Critic algorithm (ZOAC) that unifies timestep-wise parameter space perturbation, first-order policy evaluation and zeroth-order improvement into an on-policy actor-critic architecture.