A Work in Progress: Tighter Bounds on the Information Bottleneck for Deep Learning
Keywords: Deep Neural Networks, Information Bottleneck, Variational Approximations, Adversarial Robustness, Information Theoretic Tools
TL;DR: A new and empirically tighter tractable variational bound on the Information Bottleneck
Abstract: The field of Deep Neural Nets (DNNs) is still evolving and new architectures are emerging to better extract information from available data. The Information Bottleneck, IB, offers an optimal information theoretic framework for data modeling. However, IB is intractable in most settings. In recent years attempts were made to combine deep learning with IB both for optimization and to explain the inner workings of deep neural nets. VAE inspired variational approximations such as VIB became a popular method to approximate bounds on the required mutual information computations. This work continues this direction by introducing a new tractable variational upper bound for the IB functional which is empirically tighter than previous bounds. When used as an objective function it enhances the performance of previous IB-inspired DNNs in terms of test accuracy and robustness to adversarial attacks across several challenging tasks. Furthermore, the utilization of information theoretic tools allows us to analyze experiments and confirm theoretical predictions in real world problems.
Submission Number: 7
Loading