Boundary between noise and information applied to filtering neural network weight matrices

Max Staats, Matthias Thamm, Bernd Rosenow

Keywords: RMT, Noise, Filtering

TL;DR: We identify a boundary between noise and information in weight matrices of neural networks and improve the network by removing the noise part.

Abstract:

Deep neural networks have been successfully applied to a broad range of problems where overparametrization yields weight matrices which are partially random. A comparison of weight matrix
singular vectors to the Porter-Thomas distribution suggests that there is a boundary between randomness and learned information in the singular value spectrum. Inspired by this finding, we
introduce an algorithm for noise filtering, which both removes small singular values and reduces
the magnitude of large singular values to counteract the effect of level repulsion between the noise
and the information part of the spectrum. For networks trained in the presence of label noise, we
find that the generalization performance improves significantly due to noise filtering.

Student Paper: Yes

Submission Number: 61