The Conditional Entropy Bottleneck

Ian Fischer

Abstract: We present a new family of objective functions, which we term the Conditional Entropy Bottleneck (CEB). These objectives are motivated by the Minimum Necessary Information (MNI) criterion. We demonstrate the application of CEB to classification tasks. We show that CEB gives: well-calibrated predictions; strong detection of challenging out-of-distribution examples and powerful whitebox adversarial examples; and substantial robustness to those adversaries. Finally, we report that CEB fails to learn from information-free datasets, providing a possible resolution to the problem of generalization observed in Zhang et al. (2016).

Keywords: representation learning, information theory, uncertainty, out-of-distribution detection, adversarial example robustness, generalization, objective function

TL;DR: The Conditional Entropy Bottleneck is an information-theoretic objective function for learning optimal representations.

Data: [CIFAR-10](https://paperswithcode.com/dataset/cifar-10), [Fashion-MNIST](https://paperswithcode.com/dataset/fashion-mnist)