The Conditional Entropy BottleneckDownload PDF

27 Sep 2018 (modified: 21 Dec 2018)ICLR 2019 Conference Blind SubmissionReaders: Everyone
  • Abstract: We present a new family of objective functions, which we term the Conditional Entropy Bottleneck (CEB). These objectives are motivated by the Minimum Necessary Information (MNI) criterion. We demonstrate the application of CEB to classification tasks. We show that CEB gives: well-calibrated predictions; strong detection of challenging out-of-distribution examples and powerful whitebox adversarial examples; and substantial robustness to those adversaries. Finally, we report that CEB fails to learn from information-free datasets, providing a possible resolution to the problem of generalization observed in Zhang et al. (2016).
  • Keywords: representation learning, information theory, uncertainty, out-of-distribution detection, adversarial example robustness, generalization, objective function
  • TL;DR: The Conditional Entropy Bottleneck is an information-theoretic objective function for learning optimal representations.
24 Replies