The Conditional Entropy Bottleneck

Sep 27, 2018 Blind Submission readers: everyone Show Bibtex
  • Abstract: We present a new family of objective functions, which we term the Conditional Entropy Bottleneck (CEB). These objectives are motivated by the Minimum Necessary Information (MNI) criterion. We demonstrate the application of CEB to classification tasks. We show that CEB gives: well-calibrated predictions; strong detection of challenging out-of-distribution examples and powerful whitebox adversarial examples; and substantial robustness to those adversaries. Finally, we report that CEB fails to learn from information-free datasets, providing a possible resolution to the problem of generalization observed in Zhang et al. (2016).
  • Keywords: representation learning, information theory, uncertainty, out-of-distribution detection, adversarial example robustness, generalization, objective function
  • TL;DR: The Conditional Entropy Bottleneck is an information-theoretic objective function for learning optimal representations.
0 Replies