Weight matrices compression based on PDB model in deep neural networks

Xiaoling Wu, Junpeng Zhu, Zeng Li

TL;DR: Weight Matrix Compression with PDB Model

Abstract: Weight matrix compression has been demonstrated to effectively reduce overfitting and improve the generalization performance of deep neural networks. Compression is primarily achieved by filtering out noisy eigenvalues of the weight matrix. In this work, a novel **Population Double Bulk (PDB) model** is proposed to characterize the eigenvalue behavior of the weight matrix, which is more general than the existing Population Unit Bulk (PUB) model. Based on PDB model and Random Matrix Theory (RMT), we have discovered a new **PDBLS algorithm** for determining the boundary between noisy eigenvalues and information. A **PDB Noise-Filtering algorithm** is further introduced to reduce the rank of the weight matrix for compression. Experiments show that our PDB model fits the empirical distribution of eigenvalues of the weight matrix better than the PUB model, and our compressed weight matrices have lower rank at the same level of test accuracy. In some cases, our compression method can even improve generalization performance when labels contain noise. The code is avaliable at https://github.com/xlwu571/PDBLS.

Lay Summary: Deep neural networks can overfit when their weight matrices contain too much noise. A common solution is to compress these matrices by removing noisy eigenvalues. However, existing models that guide this process often fail to match real eigenvalue distributions. We propose a new model, Population Double Bulk (PDB), which better captures the eigenvalue structure. Based on this, we develop PDBLS, an algorithm to detect and remove noise-dominated components. Our method reduces the matrix rank while preserving accuracy, and can even improve generalization when labels are noisy. This provides a more effective and robust approach to weight matrix compression in deep learning. The approach can be readily applied to various network architectures.

Link To Code: https://github.com/xlwu571/PDBLS

Primary Area: Probabilistic Methods->Spectral Methods

Keywords: Weight Matrix Compression, Noise-filtering, Generalization Performance, Random Matrix Theory

Submission Number: 1722