A Unified Perspective on Value Backup and Exploration in Monte-Carlo Tree Search

Tuan Quang Dam, Carlo D'Eramo, Joni Pajarinen, Jan Peters

Keywords: Monte-Carlo Tree Search, Exploration-Exploitation, Entropy Regularization, Alpha Divergence

Abstract: Monte-Carlo Tree Search (MCTS) is a class of methods for solving complex decision-making problems through the synergy of Monte-Carlo planning and Reinforcement Learning (RL). The highly combinatorial nature of the problems commonly addressed by MCTS requires the use of efficient exploration strategies for navigating the planning tree and quickly convergent value backup methods. These crucial problems are particularly evident in recent advances that combine MCTS with deep neural networks for function approximation. In this work, we introduce a mathematical framework based on using the $\alpha$-divergence for backup and exploration in MCTS. We show that this theoretical formulation unifies different approaches, including our newly introduced ones (Power-UCT and E3W), under the same mathematical framework, allowing us to obtain different methods by simply changing the value of $\alpha$. In practice, our unified perspective offers a flexible way to balance exploration and exploitation by tuning the single $\alpha$ parameter according to the problem at hand. We validate our methods through a rigorous empirical study of a basic toy task Synthetic Tree problem.