Optimal Regret Bounds for Policy Optimization in Contextual Bandits

Orin Levy, Yishay Mansour

Track: Research Track

Keywords: Contextual Bandits, Regret, Policy Optimization

Abstract:

We present the first high-probability optimal regret bound for a policy optimization technique applied to the problem of stochastic contextual multi-armed bandit (CMAB) with general function approximation. Our algorithm is both efficient and achieves an optimal regret bound of $\widetilde{O}(\sqrt{ K|\mathcal{A}|\log|\mathcal{F}|})$, where $K$ is the number of rounds, $\mathcal{A}$ is the set of arms, and $\mathcal{F}$ is the function class used to approximate the losses.
Our results bridge the gap between theory and practice, demonstrating that the widely used policy optimization methods for the contextual bandits problem can achieve a rigorously-proved optimal regret bound.
We support our theoretical results with an empirical evaluation of our algorithm.

Submission Number: 9