Sharp Gap-Dependent Variance-Aware Regret Bounds for Tabular MDPs

Shulun Chen, Runlong Zhou, Zihan Zhang, Maryam Fazel, Simon Shaolei Du

Track: Research Track

Keywords: MDP, gap-dependent, variance

Abstract:

We consider gap-dependent regret bounds for episodic MDPs.
We show that the Monotonic Value Propagation (MVP) algorithm (\cite{zhang2024settling}) achieves a variance-aware gap-dependent regret bound of
$$\tilde{O}\left(\left(\sum_{\Delta_h(s,a)>0} \frac{H^2 \log K \land\mathtt{Var}^{\textup{c}} _ {\min}}{\Delta_h(s,a)} +\sum_{\Delta_h(s,a)=0}\frac{ H^2 \land \mathtt{Var}^{\textup{c}} _ {\min}}{\Delta_{\mathrm{min}}} + SAH^4 (S \lor H) \right) \log K\right),$$
where $H$ is the planning horizon, $S$ is the number of states, $A$ is the number of actions, and $K$ is the number of episodes. Here, $\Delta_h(s,a) =V_h^* (a) - Q_h^* (s, a)$ represents the suboptimality gap and $\Delta_{\mathrm{min}} := \min_{\Delta_h (s,a) > 0} \Delta_h(s,a)$.
The term $\mathtt{Var}^{\textup{c}}_ {\min}$ denotes the maximum conditional total variance, calculated as the maximum over all $(\pi, h, s)$ tuples of the expected total variance under policy $\pi$ conditioned on trajectories visiting state $s$ at step $h$.
$\mathtt{Var}^{\textup{c}}_ {\min}$ characterizes the maximum randomness encountered when learning any $(h, s)$ pair.
Our result stems from a novel analysis of the weighted sum of the suboptimality gap and can be potentially adapted for other algorithms.
To complement the study, we establish a lower bound of
$$\Omega \left( \sum_{\Delta_h(s,a)>0} \frac{H^2 \land \mathtt{Var}^{\textup{c}}_ {\min}}{\Delta_h(s,a)}\cdot \log K\right),$$
demonstrating the necessity of dependence on $\mathtt{Var}^{\textup{c}}_ {\min}$ even when the maximum unconditional total variance (without conditioning on $(h, s)$) approaches zero.

Submission Number: 134