On Predictive Information Sub-optimality of RNNs

Zhe Dong, Deniz Oktay, Ben Poole, Alexander A. Alemi

Abstract: Certain biological neurons demonstrate a remarkable capability to optimally compress the history of sensory inputs while being maximally informative about the future. In this work, we investigate if the same can be said of artificial neurons in recurrent neural networks (RNNs) trained with maximum likelihood. In experiments on two datasets, restorative Brownian motion and a hand-drawn sketch dataset, we find that RNNs are sub-optimal in the information plane. Instead of optimally compressing past information, they extract additional information that is not relevant for predicting the future. Overcoming this limitation may require alternative training procedures and architectures, or objectives beyond maximum likelihood estimation.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:1910.09578/code)