High-Dimensional Unbiased Prediction for Sequential Decision Making
Keywords: unbiased estimation, fairness, uncertainty estimation, swap regret, subsequence regret, extensive form games, online combinatorial optimization
TL;DR: An online algorithm for producing conditionally unbiased vector predictions that help optimize over large downstream action spaces
Abstract: We study the problem of making predictions of an adversarially chosen high-dimensional state that are \emph{unbiased} subject to an arbitrary collection of conditioning events, with the goal of tailoring these events to downstream decision makers. We give efficient algorithms for solving this problem, along with several applications that stem from choosing an appropriate set of conditioning events.
For example, we can efficiently produce predictions targeted at any polynomial number of decision makers, such that if they best respond to our predictions, each of them will have diminishing swap regret at the optimal rate. We then generalize this to the online combinatorial optimization problem, where the decision makers have large action spaces corresponding to structured subsets of a set of base actions: We give the first algorithms that can guarantee (to any polynomial number of decision makers) no regret to the best fixed action, not just overall, but on any polynomial number of \emph{subsequences} that can depend on the actions chosen as well as any external context. We show how playing in an extensive-form game can be cast into this framework, and use these results to give efficient algorithms for obtaining no \emph{subsequence regret} in extensive-form games. This gives a new family of regret guarantees that captures and generalizes previously studied notions such as regret to informed causal deviations, and is generally incomparable to other known families of efficiently obtainable guarantees.
We then turn to uncertainty quantification in machine learning, and consider the problem of producing \emph{prediction sets} for online adversarial multiclass and multilabel classification. We show how to produce class scores that have \emph{transparent coverage guarantees}: they can be used to produce prediction sets covering the true labels at the same rate as they would \emph{had our scores been the true conditional class probabilities}. We then show that these transparent coverage guarantees imply strong online adversarial \emph{conditional validity} guarantees (including \emph{set-size conditional} coverage and \emph{multigroup-fair} coverage) for (potentially \emph{multiple}) downstream prediction set algorithms relying on our class scores. Moreover, we show how to guarantee that our class scores have improved $L_2$ loss (or cross-entropy loss, or more generally any separable Bregman loss) compared to any collection of benchmark models. This can be viewed as a high-dimensional, real-valued version of \emph{omniprediction}. Compared to conformal prediction techniques, our uncertainty quantification framework gives increased flexibility and eliminates the need to choose a non-conformity score.
Submission Number: 96
Loading