TOWARDS UNDERSTANDING IN-DISTRIBUTION AND OUT-OF-DISTRIBUTION OF DEEP LEARNING WITH DEEP GENERATIVE MODELS

Glory Rongyu CHEN, Angela Yao

Abstract:

Although deep learning techniques have been developing rapidly in recent
years, we still lack an understanding of their deployment. Reasonable gener-
alization to only in-distribution (ID) distributions of deep learning models
requires understanding and recognition of ID and Out-of-Distribution
(OoD). Deep generative models have made a hit in modeling the underlying
distribution and generating samples from it. Compared with other OoD
detection methods such as discriminative and heuristic ones, the motivation
and intuition of generative-based OoD detection methods are clear and
convincing in high dimensions like image space. It only needs to threshold
likelihood produced by Deep Generative Models (DGMs) to detect OoD
data. However, researchers recently found DGMs counter-intuitively as-
sign higher likelihood to OoD data. They discovered domain priors and
model inductive biases to account for the phenomenon and came up with
calibrations to the paradigm.
In this work, we found most of the existing work adopt weak generative
backbones like VAEs and Flows. Weak DGMs have difficulty modeling
underlying distributions and thus generate poor-quality images. Using
these backbones to do OoD detection quite deviates from the motivation.
Different from them, we experiment with strong generative models and
reveal the benefits brought by the change, including significant alleviation
of the problem without the need for any calibrations. This leads to a clear
pipeline. Additionally, with the observation that DGMs do generate only
ID data, we think they are knowledgeable of what ID is. We propose
another manner of mining ID/OoD knowledge of the model which does
not rely on likelihood. We believe this will encourage the community to
think more about OoD detection using deep generative models.