Open Peer Review. Open Publishing. Open Access. Open Discussion. Open Directory. Open Recommendations. Open API. Open Source.
Information Dropout: learning optimal representations through noise
Alessandro Achille, Stefano Soatto
Nov 04, 2016 (modified: Jan 10, 2017)ICLR 2017 conference submissionreaders: everyone
Abstract:We introduce Information Dropout, a generalization of dropout that is motivated by the Information Bottleneck principle and highlights the way in which injecting noise in the activations can help in learning optimal representations of the data. Information Dropout is rooted in information theoretic principles, it includes as special cases several existing dropout methods, like Gaussian Dropout and Variational Dropout, and, unlike classical dropout, it can learn and build representations that are invariant to nuisances of the data, like occlusions and clutter. When the task is the reconstruction of the input, we show that the information dropout method yields a variational autoencoder as a special case, thus providing a link between representation learning, information theory and variational inference. Our experiments validate the theoretical intuitions behind our method, and we find that information dropout achieves a comparable or better generalization performance than binary dropout, especially on smaller models, since it can automatically adapt the noise to the structure of the network, as well as to the test sample.
TL;DR:We introduce Information Dropout, an information theoretic generalization of dropout that highlights how injecting noise can help in learning invariant representations.
Keywords:Theory, Deep learning
Enter your feedback below and we'll get back to you as soon as possible.