- TL;DR: We extend the information bottleneck method to the unsupervised multiview setting and show state of the art results on standard datasets
- Abstract: The information bottleneck method provides an information-theoretic method for representation learning, by training an encoder to retain all information which is relevant for predicting the label, while minimizing the amount of other, superfluous information in the representation. The original formulation, however, requires labeled data in order to identify which information is superfluous. In this work, we extend this ability to the multi-view unsupervised setting, in which two views of the same underlying entity are provided but the label is unknown. This enables us to identify superfluous information as that which is not shared by both views. A theoretical analysis leads to the definition of a new multi-view model that produces state-of-the-art results on the Sketchy dataset and on label-limited versions of the MIR-Flickr dataset. We also extend our theory to the single-view setting by taking advantage of standard data augmentation techniques, empirically showing better generalization capabilities when compared to traditional unsupervised approaches for representation learning.
- Code: https://github.com/mfederici/Multi-View-Information-Bottleneck
- Keywords: Information Bottleneck, Multi-View Learning, Representation Learning, Information Theory
- Original Pdf: pdf