A Unifying Post-Processing Framework for Multi-Objective Learn-to-Defer Problems
Keywords: Learn-to-Defer, Human-AI Teaming, Neyman-Pearson Lemma, Hypothesis Testing, Bayes Optimal, PAC Generalization
TL;DR: Here, we obtain an optimal solution to multi-objective learn-to-defer problem by generalizing Neyman-Pearson lemma to multi-hypothesis setting
Abstract: Learn-to-Defer is a paradigm that enables learning algorithms to work not in isolation but as a team with human experts. In this paradigm, we permit the system to defer a subset of its tasks to the expert. Although there are currently systems that follow this paradigm and are designed to optimize the accuracy of the final human-AI team, the general methodology for developing such systems under a set of constraints (e.g., algorithmic fairness, expert intervention budget, defer of anomaly, etc.) remains largely unexplored. In this paper, using a d-dimensional generalization to the fundamental lemma of Neyman and Pearson (d-GNP), we obtain the Bayes optimal solution for learn-to-defer systems under various constraints. Furthermore, we design a generalizable algorithm to estimate that solution and apply this algorithm to the COMPAS, Hatespeech, and ACSIncome datasets. Our algorithm shows improvements in terms of constraint violation over a set of learn-to-defer baselines and can control multiple constraint violations at once. The use of d-GNP is beyond learn-to-defer applications and can potentially obtain a solution to decision-making problems with a set of controlled expected performance measures.
Primary Area: Learning theory
Submission Number: 18083
Loading