Dynamic Low-rank Estimation for Transformer-based Language Models

Ting Hua, Xiao Li, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, Hongxia Jin

Submission Type: Regular Long Paper

Submission Track: Machine Learning for NLP

Keywords: low-rank estimation; matrix factorization;

Abstract:

Matrix decomposition methods, such as Singular Value Decomposition (SVD) and its importance-weighted variants, have been widely used for compressing Transformer-based language models. While importance-weighted decomposition methods alleviate the strong assumption of equal importance for each parameter in SVD, they still rely on two fundamental assumptions: 1) unchanged importance distribution during further fine-tuning, 2) equal importance across weight matrices in different layers.
Furthermore, these methods necessitate a well-trained task-specific model as the starting point and require additional fine-tuning after compression.
In this work, we proposed RankDyna, a matrix decomposition method that enables dynamic rank resource allocation among matrices across different layers during the training process.
Starting from a general pre-trained model, RankDyna accomplishes the dual goals of compression and adaptation to the downstream task, all within a single round of fine-tuning.
The extensive evaluations demonstrate that RankDyna can outperform current SOTA methods under various parameter budget levels, and the advantage of RankDyna is further enhanced with higher compression rates.

Submission Number: 4368