Free Probability for predicting the performance of feed-forward fully connected neural networks

Reda CHHAIBI, Tariq Daouda, Ezechiel Kahn

Keywords: Infinite width regime, Lyapounov exponents, Singular spectrum of Jacobians of neural networks, Stability, Free Probability Theory, Numerical Methods, Newton-Raphson method

TL;DR: Stability (and hence performance) of NNs can be probed before training thanks to Free Probability Theory, which gives a computable metamodel in the infinite width regime.

Abstract:

Gradient descent during the learning process of a neural network can be subject to many instabilities. The spectral density of the Jacobian is a key component for analyzing stability. Following the works of Pennington et al., such Jacobians are modeled using free multiplicative convolutions from Free Probability Theory (FPT).

We present a reliable and very fast method for computing the associated spectral densities, for given architecture and initialization. This method has a controlled and proven convergence. Our technique is based on an homotopy method: it is an adaptative Newton-Raphson scheme which chains basins of attraction. We find contiguous lilypad-like basins and step from one to the next, heading towards the objective.

In order to demonstrate the relevance of our method we show that the relevant FPT metrics computed before training are highly correlated to final test losses – up to 85%. We also give evidence that a very desirable feature for neural networks is the hyperbolicity of their Jacobian at initialization, while remaining at the edge of chaos.