Pseudo-Calibration: Improving Predictive Uncertainty Estimation in Domain Adaptation
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Domain Adaptation; Predictive Uncertainty; Model Calibration
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
TL;DR: We introduce Pseudo-Calibration, a novel and versatile post-hoc framework for calibrating predictive uncertainty in unsupervised domain adaptation.
Abstract: Unsupervised domain adaptation (UDA) has seen significant efforts to enhance model accuracy for an unlabeled target domain with the help of one or more labeled source domains. However, UDA models often exhibit poorly calibrated predictive uncertainty of target data, a problem that remains under-explored and poses risks in safety-critical UDA applications. The two primary challenges in addressing predictive uncertainty calibration in UDA are the absence of labeled target data and severe distribution shifts between the two domains. Traditional supervised calibration methods like \emph{temperature scaling} are inapplicable due to the former challenge. Recent studies address the first challenge by employing \emph{importance-weighting} with labeled source data but still suffer from the second challenge and additional complex density modeling. We propose Pseudo-Calibration (PseudoCal), a novel post-hoc calibration framework. In contrast to prior approaches, we consider UDA calibration as a target domain-specific unsupervised problem rather than a \emph{covariate shift} problem across domains. Our innovative use of inference-stage \emph{mixup} and \emph{cluster assumption} guarantees that a synthesized labeled pseudo-target set captures the structure of the real target. In this way, we turn the unsupervised calibration problem into a supervised one, readily solvable with \emph{temperature scaling}. Extensive empirical evaluation across 5 diverse UDA scenarios involving 10 UDA methods consistently demonstrates the superior performance of PseudoCal over alternative calibration methods.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 516
Loading