Unsupervised Temperature Scaling: Robust Post-processing Calibration for Domain Shift

Azadeh Sadat Mozafari, Hugo Siqueira Gomes, Christian Gagne

Keywords: calibration, domain shift, uncertainty prediction, deep neural networks, temperature scaling

TL;DR: A robust post-processing calibration method for domain shift.

Abstract: The uncertainty estimation is critical in real-world decision making applications, especially when distributional shift between the training and test data are prevalent. Many calibration methods in the literature have been proposed to improve the predictive uncertainty of DNNs which are generally not well-calibrated. However, none of them is specifically designed to work properly under domain shift condition. In this paper, we propose Unsupervised Temperature Scaling (UTS) as a robust calibration method to domain shift. It exploits test samples to adjust the uncertainty prediction of deep models towards the test distribution. UTS utilizes a novel loss function, weighted NLL, that allows unsupervised calibration. We evaluate UTS on a wide range of model-datasets which shows the possibility of calibration without labels and demonstrate the robustness of UTS compared to other methods (e.g., TS, MC-dropout, SVI, ensembles) in shifted domains.

Code: https://github.com/uts-iclr2020/uts-iclr2020