Directional Ensemble Aggregation for Actor-Critics

Nicklas Werge, Yi-Shan Wu, Bahareh Tasdighi, Melih Kandemir

Keywords: reinforcement learning, off-policy, actor-critic, adaptive ensemble learning

TL;DR: Directional Ensemble Aggregation (DEA) is a fully learnable actor-critic method that adaptively balances conservatism and exploration by aggregating $Q$-values based on ensemble disagreement.

Abstract: Reliable $Q$-value estimation is central to off-policy reinforcement learning in continuous control. Standard actor-critic methods often address overestimation bias by aggregating ensembles of $Q$-values conservatively, for example by taking their minimum. While effective at reducing bias, these static rules discard useful information, cannot adapt to training dynamics, and generalize poorly across learning regimes. We propose Directional Ensemble Aggregation (DEA), a fully learnable aggregation method that replaces static aggregation with a dynamic mechanism capable of interpolating between conservative and explorative strategies as training progresses. DEA introduces two learnable directional parameters, one regulating critic conservatism and the other guiding actor exploration. Both are learned using disagreement-weighted Bellman errors, where updates depend only on the sign of each sample’s error. This decoupled design allows DEA to adjust automatically to task-specific uncertainty, ensemble size, and update frequency in a data-driven manner. Empirically, DEA generalizes across MuJoCo and DeepMind Control Suite benchmarks in both interactive and sample-efficient learning regimes.

Primary Area: reinforcement learning

Submission Number: 628