Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models

Tianxiang Gao, Xiaokai Huo, Hailiang Liu, Hongyang Gao

Keywords: Gradient descent, deep equilibrium model, Gaussian processes, kernel methods, NNGP, NTK

TL;DR: We show deep equilibrium model as a infinite-depth neural network with shared weight is equivalent to a non-degenerate Gaussian process in the limit of width approaching to infinite

Abstract:

Neural networks with wide layers have attracted significant attention due to their equivalence to Gaussian processes, enabling perfect fitting of training data while maintaining generalization performance, known as benign overfitting. However, existing results mainly focus on shallow or finite-depth networks, necessitating a comprehensive analysis of wide neural networks with infinite-depth layers, such as neural ordinary differential equations (ODEs) and deep equilibrium models (DEQs).

In this paper, we specifically investigate the deep equilibrium model (DEQ), an infinite-depth neural network with shared weight matrices across layers. Our analysis reveals that as the width of DEQ layers approaches infinity, it converges to a Gaussian process, establishing what is known as the Neural Network and Gaussian Process (NNGP) correspondence. Remarkably, this convergence holds even when the limits of depth and width are interchanged, which is not observed in typical infinite-depth Multilayer Perceptron (MLP) networks. Furthermore, we demonstrate that the associated Gaussian vector remains non-degenerate for any pairwise distinct input data, ensuring a strictly positive smallest eigenvalue of the corresponding kernel matrix using the NNGP kernel. These findings serve as fundamental elements for studying the training and generalization of DEQs, laying the groundwork for future research in this area.

Submission Number: 5423