Learning to Defer with an Uncertain Rejector via Conformal Prediction

TMLR Paper6288 Authors

Abstract: Learning to defer (L2D) allows prediction tasks to be allocated to a human or machine decision maker, thus getting the best of both’s abilities. This allocation decision crucially depends on a ‘rejector’ function. In practice, the rejector could be poorly fit or otherwise misspecified. In this work, we perform uncertainty quantification for the rejector sub-component of the L2D framework. We use conformal prediction to allow the rejector to output prediction sets or intervals of a user-defined confidence level (with distribution-free guarantees), instead of just the binary outcome of ‘defer’ or not. On tasks ranging from image to hate speech classification, we demonstrate that the uncertainty in the rejector translates to safer decisions via two forms of selective prediction

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Manuel_Haussmann1

Submission Number: 6288