Random matrix theory analysis of neural network weight matrices

Matthias Thamm, Max Staats, Bernd Rosenow

Keywords: Random matrix theory, RMT, neural networks

TL;DR: We use random matrix theory to locate information in neural network weights.

Abstract: As neural network weight matrices are initialized randomly, they conform precisely to random matrix theory (RMT) predictions before training. Post-training, deviations from RMT predictions indicate task-specific information encoded in the weights. We analyze feedforward and convolutional neural network weights trained on image recognition tasks. We demonstrate that most of the weights' singular values follow universal RMT predictions even after training, suggesting that major parts of weights remain random. By comparing singular value spectra with the Marchenko-Pastur distribution and singular vector entries with the Porter-Thomas distribution, we identify significant deviations only in the parts associated with the largest singular values. We argue that a comparison to RMT predictions allows locating learned information in the weights. In addition, the RMT analysis enables us to differentiate between networks trained within various learning regimes.

Student Paper: No

Submission Number: 62