Information Regularized Neural Networks

Tianchen Zhao, Dejiao Zhang, Zeyu Sun, Honglak Lee

Abstract: We formulate an information-based optimization problem for supervised classification. For invertible neural networks, the control of these information terms is passed down to the latent features and parameter matrix in the last fully connected layer, given that mutual information is invariant under invertible map. We propose an objective function and prove that it solves the optimization problem. Our framework allows us to learn latent features in an more interpretable form while improving the classification performance. We perform extensive quantitative and qualitative experiments in comparison with the existing state-of-the-art classification models.

Keywords: supervised classification, information theory, deep learning, regularization

TL;DR: we propose a regularizer that improves the classification performance of neural networks