Rethinking Test-time Likelihood: The Likelihood Path Principle and Its Application to OOD Detection

Sicong Huang, Jiawei He, Yik Chau Lui

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: outlier detection, ood, out-of-distribution, anomaly detection, variational autoencoder, VAE

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: By leveraging sufficient statistics from the likelihood path (LPath) of VAEs, our mathematically provable method delivers SOTA performance for unsupervised, one sample OOD detection

Abstract:

While likelihood is attractive in theory, its estimates by deep generative models (DGMs) are often broken in practice, and perform poorly for OOD Detection. Various recent works started to consider alternative summary statistics and achieved better performances. However, such recipes do not come with provable guarantees, nor is it clear that their choices extract sufficient information.

We attempt to change this by conducting a case study on variational autoencoders (VAEs). First, we introduce the *likelihood path (LPath) principle*, generalizing the likelihood principle. This narrows the search for informative summary statistics down to the *minimal sufficient statistics* of VAEs' conditional likelihoods. Second, introducing new theoretic tools such as *essential support*, *essential distance* and *co-Lipschitzness*, we obtain non-asymptotic provable OOD detection guarantees for certain distillation of the minimal sufficient statistics. The corresponding LPath algorithm demonstrates SOTA performances, even using simple and small VAEs with poor likelihood estimates. To our best knowledge, this is the first provable unsupervised OOD method that delivers excellent empirical results, better than any other VAEs based techniques.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8717