EBMaC: Empirical Bayes and Matrix Constraints for Label Shift

Mushan Li, Kihyun Han, YANYUAN MA

TL;DR: This paper presents EBMaC, a hierarchical model for label shift estimation using the empirical Bayes method and matrix constraints to enhance model performance.

Abstract: We estimate the importance weights and their associated confidence set in label shift problems using hierarchical models via the Empirical Bayes and Matrix Constraints (EBMaC) method. Our approach accommodates dispersion beyond what is permitted by the classic multinomial model and produces exact confidence regions in finite samples for confusion matrix and predicted labels. In addition, we describe the dependence structure of the importance weights in matrix constraints. Through a linear programming technique, we are able to compute smaller confidence sets and shorter elementwise confidence intervals for importance weights compared to existing methods, while maintaining the probability guarantee. Applying the results to prediction in the target domain directly yields smaller conformal prediction set and PAC prediction set. Numerical experiments demonstrate the advantages of EBMaC in producing tighter confidence sets for the importance weights both marginally and jointly.

Primary Area: General Machine Learning

Keywords: confidence intervals, hierarchical models, label shift, linear programming, matrix constraints, overdispersion, prediction sets

Submission Number: 14397