Efficient Multivariate Bandit Algorithm with Path Planning

Keyu Nie, Zezhong Zhang, Ted Tao Yuan, Rong Song, Pauline Berry Burke

Abstract: In this paper, we solve the arms exponential exploding issues in multivariate Multi-Armed Bandit (Multivariate-MAB) problem when the arm dimension hierarchy is considered. We propose a framework called path planning (TS-PP) which utilizes decision graph/trees to model arm reward success rate with m-way dimension interaction, and adopts Thompson sampling (TS) for heuristic search of arm selection. Naturally, it is quite straightforward to combat the curse of dimensionality using a serial processes that operates sequentially by focusing on one dimension per each process. For our best acknowledge, we are the first to solve Multivariate-MAB problem using graph path planning strategy and deploying alike Monte-Carlo tree search ideas. Our proposed method utilizing tree models has advantages comparing with traditional models such as general linear regression. Simulation studies validate our claim by achieving faster convergence speed, better efficient optimal arm allocation and lower cumulative regret.

Keywords: Multivariate Multi-armed Bandit, Monte Carlo Tree Search, Thompson Sampling, Path Planning

TL;DR: A novel way utilizing tree models to solve multivariate Multi-Armed Bandit problem.