Wider Networks Learn Better Features

Dar Gilboa, Guy Gur-Ari

TL;DR: We visualize the hidden states of wide networks, finding that they contain more information about the inputs than narrow networks with equal performance, and show that wide networks fine-tuned to perform novel tasks outperform narrow networks.

Abstract: Transferability of learned features between tasks can massively reduce the cost of training a neural network on a novel task. We investigate the effect of network width on learned features using activation atlases --- a visualization technique that captures features the entire hidden state responds to, as opposed to individual neurons alone. We find that, while individual neurons do not learn interpretable features in wide networks, groups of neurons do. In addition, the hidden state of a wide network contains more information about the inputs than that of a narrow network trained to the same test accuracy. Inspired by this observation, we show that when fine-tuning the last layer of a network on a new task, performance improves significantly as the width of the network is increased, even though test accuracy on the original task is independent of width.

Keywords: Interpretability, transfer learning

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1909.11572/code)