Towards Well-Calibrated Active Learning

TMLR Paper9031 Authors

Abstract: We study well-calibrated Active Learning (AL), i.e., the problem of actively learning a classifier with a low calibration error. One of the most popular Acquisition Functions (AFs) in pool-based AL is querying by the model's uncertainty. However, we recognize that an uncalibrated uncertainty model on the unlabeled pool may significantly affect AF effectiveness, leading to high calibration error and sub-optimal generalization on unseen data. Deep Neural Networks (DNNs) make the problem even worse, as model uncertainty from DNNs is usually uncalibrated. Therefore, we propose a new AF, Calibration Priority Sampling, by estimating calibration errors and query samples with the highest calibration error before leveraging DNN uncertainty. Specifically, we utilize a kernel calibration error estimator under the covariate shift and formally show that AL with this AF eventually leads to a bounded calibration error on the unlabeled pool and unseen test data. Empirically, our proposed method surpasses other AF baselines by having a lower calibration and generalization error across pool-based AL settings.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Chao_Chen1

Submission Number: 9031