Continuous Control With Ensemble Deep Deterministic Policy Gradients
Keywords: deep learning, reinforcement learning, ensemble learning, MuJoCo, continuous control, stable performance, deterministic policy gradient
Abstract: The growth of deep reinforcement learning (RL) has brought multiple exciting tools and methods to the field. This rapid expansion makes it important to understand the interplay between individual elements of the RL toolbox. We approach this task from an empirical perspective by conducting a study in the continuous control setting. We present multiple insights of fundamental nature, including: a commonly used additive action noise is not required for effective exploration and can even hinder training; the performance of policies trained using existing methods varies significantly across training runs, epochs of training, and evaluation runs; the critics' initialization plays the major role in ensemble-based actor-critic exploration, while the training is mostly invariant to the actors' initialization; a strategy based on posterior sampling explores better than the approximated UCB combined with the weighted Bellman backup; the weighted Bellman backup alone cannot replace the clipped double Q-Learning. As a conclusion, we show how existing tools can be brought together in a novel way, giving rise to the Ensemble Deep Deterministic Policy Gradients (ED2) method, to yield state-of-the-art results on continuous control tasks from \mbox{OpenAI Gym MuJoCo}. From the practical side, ED2 is conceptually straightforward, easy to code, and does not require knowledge outside of the existing RL toolbox.
One-sentence Summary: We conduct an empirical analysis of multiple tools from the RL toolbox in the continuous control setting and propose Ensemble Deep Deterministic Policy Gradients (ED2), which achieves state-of-the-art performance.
Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2111.15382/code)
15 Replies
Loading