Dual Representation Learning for Out-of-distribution Detection

Zhilin Zhao, Longbing Cao

Abstract: To classify in-distribution samples, deep neural networks explore strongly label-related information and discard weakly label-related information according to the information bottleneck. Out-of-distribution samples drawn from distributions differing from that of in-distribution samples could be assigned with unexpected high-confidence predictions because they could obtain minimum strongly label-related information. To distinguish in- and out-of-distribution samples, Dual Representation Learning (DRL) makes out-of-distribution samples harder to have high-confidence predictions by exploring both strongly and weakly label-related information from in-distribution samples. For a pretrained network exploring strongly label-related information to learn label-discriminative representations, DRL trains its auxiliary network exploring the remaining weakly label-related information to learn distribution-discriminative representations. Specifically, for a label-discriminative representation, DRL constructs its complementary distribution-discriminative representation by integrating diverse representations less similar to the label-discriminative representation. Accordingly, DRL combines label- and distribution-discriminative representations to detect out-of-distribution samples. Experiments show that DRL outperforms the state-of-the-art methods for out-of-distribution detection.

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

In this revision, we have made significant changes to address all comments and substantially enhance the paper readership. In short, the main changes are summarized below. We have:

1) provided deeper analyses for the experimental results;
2) discussed the advantages and limitations of the information bottleneck principle;
3) run a set of ablation study to verify that the label- and distribution-discriminative representations are complementary;
4) provided more explanations for the equations;
5) considered more state-of-the-art compared methods in literature review and experiments;
6) evaluated performance on near and far OOD datasets;
7) presented the visualization results on Mini-ImageNet; and
8) arranged a thorough proofreading of the paper.

Code: https://github.com/Lawliet-zzl/DRL

Assigned Action Editor: ~Matthew_Blaschko1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 1114