Robustness and/or Redundancy Emerge in Overparametrized Deep Neural Networks

Stephen Casper, Xavier Boix, Vanessa D'Amario, Christopher Rodriguez, Ling Guo, Kasper Vinken, Gabriel Kreiman

TL;DR: Probing robustness and redundancy in deep neural networks reveals capacity-constraining features which help to explain non-overfitting.

Abstract: Deep neural networks (DNNs) perform well on a variety of tasks despite the fact that most used in practice are vastly overparametrized and even capable of perfectly fitting randomly labeled data. Recent evidence suggests that developing "compressible" representations is key for adjusting the complexity of overparametrized networks to the task at hand and avoiding overfitting (Arora et al., 2018; Zhou et al., 2018). In this paper, we provide new empirical evidence that supports this hypothesis, identifying two independent mechanisms that emerge when the network’s width is increased: robustness (having units that can be removed without affecting accuracy) and redundancy (having units with similar activity). In a series of experiments with AlexNet, ResNet and Inception networks in the CIFAR-10 and ImageNet datasets, and also using shallow networks with synthetic data, we show that DNNs consistently increase either their robustness, their redundancy, or both at greater widths for a comprehensive set of hyperparameters. These results suggest that networks in the deep learning regime adjust their effective capacity by developing either robustness or redundancy.

Keywords: overparametrized dnns, robustness, redundancy, compressibility, generalization