Dual Representation Learning for Out-of-distribution Detection

Published: 21 Aug 2023, Last Modified: 21 Aug 2023Accepted by TMLREveryoneRevisionsBibTeX
Abstract: To classify in-distribution samples, deep neural networks explore strongly label-related information and discard weakly label-related information according to the information bottleneck. Out-of-distribution samples drawn from distributions differing from that of in-distribution samples could be assigned with unexpected high-confidence predictions because they could obtain minimum strongly label-related information. To distinguish in- and out-of-distribution samples, Dual Representation Learning (DRL) makes out-of-distribution samples harder to have high-confidence predictions by exploring both strongly and weakly label-related information from in-distribution samples. For a pretrained network exploring strongly label-related information to learn label-discriminative representations, DRL trains its auxiliary network exploring the remaining weakly label-related information to learn distribution-discriminative representations. Specifically, for a label-discriminative representation, DRL constructs its complementary distribution-discriminative representation by integrating diverse representations less similar to the label-discriminative representation. Accordingly, DRL combines label- and distribution-discriminative representations to detect out-of-distribution samples. Experiments show that DRL outperforms the state-of-the-art methods for out-of-distribution detection.
Submission Length: Long submission (more than 12 pages of main content)
Changes Since Last Submission: In this revision, we have made significant changes to address all comments and substantially enhance the paper readership. In short, the main changes are summarized below. We have: 1) provided deeper analyses for the experimental results; 2) discussed the advantages and limitations of the information bottleneck principle; 3) run a set of ablation study to verify that the label- and distribution-discriminative representations are complementary; 4) provided more explanations for the equations; 5) considered more state-of-the-art compared methods in literature review and experiments; 6) evaluated performance on near and far OOD datasets; 7) presented the visualization results on Mini-ImageNet; and 8) arranged a thorough proofreading of the paper.
Code: https://github.com/Lawliet-zzl/DRL
Assigned Action Editor: ~Matthew_Blaschko1
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Number: 1114