Is the acquisition worth the cost? Surrogate losses for Consistent Two-stage Classifiers

Florence Regol, Joseph Cotnareanu, Theodore Glavas, Mark Coates

Keywords: consistency, dynamic networks, learning to defer

Abstract:

Recent years have witnessed the emergence of a spectrum of foundation models, covering a broad range of capabilities and costs. Often, we effectively use foundation models as feature generators and train classifiers that use the outputs of these models to make decisions. In this paper, we consider an increasingly relevant setting where we have two classifier stages. The first stage has access to features $x$ and has the option to make a classification decision or defer, while incurring a cost, to a second classifier that has access to features $x$ and $z$. This is similar to the ``learning to defer'' setting, with the important difference that we train both classifiers jointly, and the second classifier has access to more information. The natural loss for this setting is an $\ell_{01c}$ loss, where a penalty is paid
for incorrect classification, as in $\ell_{01}$, but an additional penalty $c$ is paid
for consulting the second classifier. The $\ell_{01c}$ loss is unwieldy
for training. Our primary contribution in this paper is the derivation
of a hinge-based surrogate loss $\ell^c_{hinge}$ that is much more
amenable to training but also satisfies the property that
$\ell^c_{hinge}$-consistency implies $\ell_{01c}$-consistency.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 22699