Explaining the Out-of-Distribution Detection Paradox through Likelihood Peaks

Hamidreza Kamkari, Brendan Leigh Ross, Jesse C. Cresswell, Anthony L. Caterini, Rahul G Krishnan, Gabriel Loaiza-Ganem

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: out-of-distribution detection, normalizing flows, manifold hypothesis, intrinsic dimension

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We show that despite failing for out-of-distribution detection on their own, densities of normalizing flows can succeed at this task if paired with an estimate of intrinsic dimension

Abstract: Likelihood-based deep generative models (DGMs) commonly exhibit a puzzling behaviour: when trained on a relatively complex dataset, they assign higher likelihood values to out-of-distribution (OOD) data from simpler sources. Adding to the mystery, OOD samples are never generated by these DGMs despite having high likelihoods. This two-pronged paradox has yet to be conclusively explained, making likelihood-based OOD detection unreliable. Our primary observation is that high-likelihood regions will not be generated if they contain minimal probability mass, which can occur if the density is sharply peaked. We demonstrate how this seeming contradiction of large densities yet low probability mass can occur on data confined to low dimensional manifolds. We also show that this scenario can be identified through local intrinsic dimension (LID) estimation, and propose a method for OOD detection which pairs the likelihoods and LID estimates obtained from a pre-trained DGM. Moreover, we provide an efficient method for estimating LID from a normalizing flow model, improving upon existing estimators, and enabling state-of-the-art OOD detection performance with respect to comparable flow-based benchmarks.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8543