On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels

Amnon Geifman, Meirav Galun, David Jacobs, Ronen Basri

Keywords: Convolutional Neural Tangent Kernel, Spectral Theory, Reproducing Kernel Hilbert Space

TL;DR: We provide a theoretical study of the spectral properties of Convolutional Neural Tangent and Gaussian Process kernels

Abstract: We study the properties of various over-parameterized convolutional neural architectures through their respective Gaussian Process and Neural Tangent kernels. We prove that, with normalized multi-channel input and ReLU activation, the eigenfunctions of these kernels with the uniform measure are formed by products of spherical harmonics, defined over the channels of the different pixels. We next use hierarchical factorizable kernels to bound their respective eigenvalues. We show that the eigenvalues decay polynomially, quantify the rate of decay, and derive measures that reflect the composition of hierarchical features in these networks. Our theory provides a concrete quantitative characterization of the role of locality and hierarchy in the inductive bias of over-parameterized convolutional network architectures.