An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithms

Li Wang, Sudun, Xingjian Zhang, wenjun wu, Lei Huang

Keywords: reinforcement learning, batch normalization, systematic analysis, off-policy actor-critic algorithms

TL;DR: We conducted a systematic analysis of the use of batch normalization in off-policy actor-critic algorithms.

Abstract: Batch Normalization (BN) has played a pivotal role in the success of deep learning by improving training stability, mitigating overfitting, and enabling more effective optimization. However, its adoption in deep reinforcement learning (DRL) has been limited due to the inherent non-i.i.d.\ nature of data and the dynamically shifting distributions induced by the agent’s learning process. In this paper, we argue that, despite these challenges, BN retains unique advantages in DRL settings, particularly through its stochasticity and its ability to ease training. When applied appropriately, BN can adapt to evolving data distributions and enhance both convergence speed and final performance. To this end, we conduct a comprehensive empirical study on the use of BN in off-policy actor-critic algorithms, systematically analyzing how different training and evaluation modes impact performance. We further identify failure modes that lead to instability or divergence, analyze their underlying causes, and propose the Mode-Aware Batch Normalization (MA-BN) method with practical actionable recommendations for robust BN integration in DRL pipelines. We also empirically validate that, in RL settings, MA-BN accelerates and stabilizes training, broadens the effective learning rate range, enhances exploration, and reduces overall optimization difficulty. Our code is available at: \url{https://anonymous.4open.science/r/bn_rl468305485}.

Primary Area: reinforcement learning

Submission Number: 5798