Optimal Posterior Sampling for Policy Identification in Tabular Markov Decision Processes

Cyrille Kone, Kevin Jamieson

TL;DR: We study the identification of the best policy in tabular MDP using cposterior sampling and online learning to guide the exploration

Abstract:

We study the $(\varepsilon, \delta)$-PAC policy identification problem in finite-horizon episodic Markov Decision Processes. Existing approaches provide finite-time guarantees for approximate settings ($\varepsilon>0$) but suffer from high computational cost, rendering them hard to implement, and also suffer from suboptimal dependence on $\log(1/\delta)$. We propose a randomized and computationally efficient algorithm for best policy identification that combines posterior sampling with an online learning algorithm to guide exploration in the MDP. Our method achieves asymptotic optimality in sample complexity, also in terms of posterior contraction rate, and runs in $O(S^2AH)$ per episode, matching standard model-based approaches. Unlike prior algorithms such as MOCA and PEDEL, our guarantees remain meaningful in the asymptotic regime and avoid sub-optimal polynomial dependence on $\log(1/\delta)$.
Our results provide both theoretical insights and practical tools for efficient policy identification in tabular MDPs.

Submission Number: 364