Revealing the Utilized Rank of Subspaces of Learning in Neural Networks
Keywords: efficiency, vision transformers, low rank decomposition
TL;DR: We propose a simple low-rank, data dependent projection of the weights of a learned network that exposes how efficiently it learns a particular dataset, and exploit it for parametric and compute efficiency.
Abstract: In this work, we study how well the learned weights of a neural network utilize the space available to them. This notion is related to capacity, but additionally incorporates the interaction of the network architecture with the dataset. Most learned weights appear to be full rank, and are therefore not amenable to low rank decomposition. This deceptively implies that the weights are utilizing the entire space available to them. We propose a simple data-driven transformation that projects the weights onto the subspace where the data and the weight interact. This preserves the functional mapping of the layer and reveals its low rank structure. In our findings, we conclude that most models utilize a fraction of the available space. For instance, for ViTB-16 and ViTL-16 trained on ImageNet, the mean layer utilization is 35\% and 20\% respectively. Our transformation results in reducing the parameters to 50\% and 25\% respectively, while resulting in less than 0.2\% accuracy drop after fine-tuning. We also show that self-supervised pre-training drives this utilization up to 70\%, justifying its suitability for downstream tasks.
Submission Number: 55
Loading