Uncertainty Quantification Using a Codebook of Encoders

Ifigeneia Apostolopoulou, Benjamin Eysenbach, Frank Nielsen, Artur Dubrawski

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: uncertainty quantification, out-of-distribution detection, information theory, information bottleneck, variational information bottleneck, clustering, deterministic uncertainty methods, bregman divergence, rate-distortion theory, compression, quantization

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Many machine learning applications are limited not by the accuracy of current models but by the inability of these models to assign confidence to their predictions – the models don’t know what they don’t know. Among methods that do provide uncertainty estimates, there remains a tradeoff between reliable yet expensive methods (e.g., deep ensembles) and lightweight alternatives that can be miscalibrated. In this paper, we propose a lightweight uncertainty quantification method with performance comparable to deep ensembles across a range of tasks and metrics. The key idea behind our approach is to revise and augment prior information bottleneck methods with a codebook to obtain a compressed representation of all inputs seen during training. Uncertainty over a new example can then be quantified by its distance from this codebook. The resulting method, the Uncertainty Aware Information Bottleneck (UA-IB), requires only a single forward pass to provide uncertainty estimates. Our experiments show that UA-IB can achieve better Out-of-Distribution (OOD) detection and calibration than prior methods, including those based on the standard information bottleneck.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5315