Decision-Focused Evaluation of Worst-Case Distribution Shift

Kevin Ren, Yewon Byun, Bryan Wilder

Keywords: model evaluation, reliability, distribution shift, resource allocation

TL;DR: Evaluation of worst-case distribution shifts over a constrained set for predictive resource allocation problems.

Abstract: Recent studies have shown that performance on downstream optimization tasks often diverges from standard accuracy-based losses, highlighting that the loss function of a predictive model should align with the decision task of the downstream optimizer. Despite this observation, no work— to our knowledge—has yet examined the impact of this divergence for distribution shift. In this paper, we demonstrate that worst-case distribution shifts identified by traditional average accuracy-based metrics fundamentally differ from those for the downstream decision task at hand. We introduce a novel framework that employs a hierarchical model structure to identify worst-case distribution shifts in predictive resource allocation settings. This task is more difficult than in standard distribution shift settings because of combinatorial interactions, where decisions depend on the joint presence of individuals in the allocation task. We show that the problem can be reformulated as a submodular optimization problem, enabling efficient approximations, to capture shifts both within and across instances of the optimization problem.

List Of Authors: Ren, Kevin and Byun, Yewon and Wilder, Bryan

Submission Number: 582