High Probability Bound for Cross-Learning Contextual Bandits with Unknown Context Distributions

Ruiyuan Huang, Zengfeng Huang

TL;DR: We give a nearly optimal high probability bound for the cross-learning contextual bandits with unknown context distributions.

Abstract: Motivated by applications in online bidding and sleeping bandits, we examine the problem of contextual bandits with cross learning, where the learner observes the loss associated with the action across all possible contexts, not just the current round’s context. Our focus is on a setting where losses are chosen adversarially, and contexts are sampled i.i.d. from a specific distribution. This problem was first studied by Balseiro et al. (2019), who proposed an algorithm that achieves near-optimal regret under the assumption that the context distribution is known in advance. However, this assumption is often unrealistic. To address this issue, Schneider & Zimmert (2023) recently proposed a new algorithm that achieves nearly optimal expected regret. It is well-known that expected regret can be significantly weaker than high-probability bounds. In this paper, we present a novel, in-depth analysis of their algorithm and demonstrate that it actually achieves near-optimal regret with high probability. There are steps in the original analysis by Schneider & Zimmert (2023) that lead only to an expected bound by nature. In our analysis, we introduce several new insights. Specifically, we make extensive use of the weak dependency structure between different epochs, which was overlooked in previous analyses. Additionally, standard martingale inequalities are not directly applicable, so we refine martingale inequalities to complete our analysis.

Lay Summary: Contextual bandits are widely used in today's industry. Assuming we can see the feedback across different contexts introduces new challenges. We analyze an algorithm for this setting and show that its performance has high-probability assurance. High-probability assurance is critical for a robust algorithm.

Primary Area: Theory->Online Learning and Bandits

Keywords: bandit, contextual bandit, cross-learning, high probability bounds

Submission Number: 15881