On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process KernelsDownload PDF

Published: 31 Oct 2022, Last Modified: 11 Jan 2023NeurIPS 2022 AcceptReaders: Everyone
Keywords: Convolutional Neural Tangent Kernel, Spectral Theory, Reproducing Kernel Hilbert Space
TL;DR: We provide a theoretical study of the spectral properties of Convolutional Neural Tangent and Gaussian Process kernels
Abstract: We study the properties of various over-parameterized convolutional neural architectures through their respective Gaussian Process and Neural Tangent kernels. We prove that, with normalized multi-channel input and ReLU activation, the eigenfunctions of these kernels with the uniform measure are formed by products of spherical harmonics, defined over the channels of the different pixels. We next use hierarchical factorizable kernels to bound their respective eigenvalues. We show that the eigenvalues decay polynomially, quantify the rate of decay, and derive measures that reflect the composition of hierarchical features in these networks. Our theory provides a concrete quantitative characterization of the role of locality and hierarchy in the inductive bias of over-parameterized convolutional network architectures.
Supplementary Material: pdf
21 Replies