Sample-Efficient Omniprediction for Proper Losses

Isaac Gibbs, Ryan Tibshirani

Abstract:

We consider the problem of constructing probabilistic predictions that lead to accurate decisions
when employed by downstream users to inform actions. For a single decision maker, designing an optimal predictor is equivalent to minimizing a proper loss function corresponding to the negative utility of
that individual. For multiple decision makers, our problem can be viewed as a variant of omniprediction
in which the goal is to design a single predictor that simultaneously minimizes multiple losses. Existing algorithms for achieving omniprediction broadly fall into two categories: 1) boosting methods that
optimize other auxiliary targets such as multicalibration and obtain omniprediction as a corollary, and
2) adversarial two-player game based approaches that estimate and respond to the “worst-case” loss in
an online fashion. We give lower bounds demonstrating that multicalibration is a strictly more difficult
problem than omniprediction and thus the former approach must incur suboptimal sample complexity.
For the latter approach, we discuss how these ideas can be used to obtain a sample-efficient algorithm
through an online-to-batch conversion. This conversion has the downside of returning a complex, randomized predictor. We improve on this method by designing a more direct, unrandomized algorithm
that exploits structural elements of the set of proper losses.