Learning to Explore With Lagrangians For Bandits Under Unknown Constraints

Udvas Das, Debabrota Basu

TL;DR: We design asymptotically optimal Pure Exploration algorithms that can identify $r$-optimal policies unknown linear constraints.

Abstract: Pure exploration in bandits formalises multiple real-world problems, such as tuning hyper-parameters or conducting user studies to test a set of items, where different safety, resource, and fairness constraints on the decision space naturally appear. We study these problems as pure exploration in multi-armed bandits with unknown linear constraints, where the aim is to identify an *$r$-optimal and feasible policy* as fast as possible with a given level of confidence. First, we propose a Lagrangian relaxation of the sample complexity lower bound for pure exploration under constraints. Second, we leverage properties of convex optimisation in the Lagrangian lower bound to propose two computationally efficient extensions of Track-and-Stop and Gamified Explorer, namely LATS and LAGEX. Then, we propose a constraint-adaptive stopping rule, and while tracking the lower bound, use optimistic estimate of the feasible set at each step. We show that LAGEX achieves asymptotically optimal sample complexity upper bound, while LATS shows asymptotic optimality up to *novel* constraint-dependent constants. Finally, we conduct numerical experiments with different reward distributions and constraints that validate efficient performance of LATS and LAGEX.

Code Dataset Promise: Yes

Code Dataset Url: https://github.com/udvasdas/Pure-exploration-with-unknown-linear-constraints/

Format Confirmation: I agree that I have read and followed the formatting instructions for the camera ready version.

Submission Number: 2312