Information in Infinite Ensembles of Infinitely-Wide Neural Networks

Ravid Shwartz-Ziv, Alexander A. Alemi

Keywords: information theory, NTK, ensemble, generalization

TL;DR: Infinite ensembles of infinitely wide neural networks are an interesting model family from an information theoretic perspective.

Abstract: In this preliminary work, we study the generalization properties of infinite ensembles of infinitely-wide neural networks. Amazingly, this model family admits tractable calculations for many information-theoretic quantities. We report analytical and empirical investigations in the search for signals that correlate with generalization.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1911.09189/code)